Nmap Security Scanner Ref Guide Security Lists Security Tools Site News Advertising About/Contact Credits Sponsors |
Intro | Reference Guide | Book | Install Guide
Download | Changelog | Zenmap GUI | Docs
Bug Reports | OS Detection | Propaganda | Related Projects
In the Movies | In the News
Implementation
| Chapter 9. Nmap Scripting Engine |
## Implementation We don't need a dozen pages of low-level trivial details, but it would be nice to have a few sections describing notable aspects of the NSE implementation (maybe things like how the parallelization algorithms work, how Lua is embedded, performance related notes. Information which might help script writers is particularly desirable. I tend to think reasons for choosing Lua may be better suited to the section called “Lua Base Language”, but it could be placed here instead. Now how does all this work? The following section describes some interesting aspects of the NSE. While the focus primarily lies on giving script writers a better feeling of what happens with scripts, it should also provide a starting point for understanding (and extending) the NSE sources. ### Initialization Phase During its initialization stage, Nmap loads the Lua interpreter and its provided libraries get loaded. These libraries are: The package library (namespace: `package`)—Lua's package-lib provides (among others) the `require` function, used to load modules from the nselib. The table library (namespace: `table`)—The table manipulation library contains many functions used to operate on `tables`—Lua's central data structure. The I/O library (namespace: `io`)—The Input/Output library offers functions such as reading files and reading the output from programs you execute. The OS library (namespace: `os`)—The Operating System lib provides facilities of the operating system, including filesystem operations (renaming/removing files, creating of temporary filenames) and access to the environment. The string library (namespace: `string`)—The string library helps you with functions used to manipulate strings inside Lua. Functions include: printf-style string formating, pattern matching using Lua-style patterns, substring extraction, etc. The math library (namespace: `math`)—Since usually numbers in Lua correspond to the `double` C-type, the math library gives you access to rounding functions, trigonometric functions, random number generation, and many more. The debug library (namespace: `debug`)—The debug library provides you with a somewhat lower level API to the Lua-interpreter. Through it you can access functions along the execution stack, get function closures and object metatables, etc. In addition to loading the libraries provided with Lua, the functions in the `nmap` namespace also get loaded. and search path for modules is set to the default one prepended by the nselib directory (which is searched in the locations Nmap searches for its data files and scripts). In this step the provided script arguments also get stored inside the `registry`. The next phase of NSE initialization is loading the chosen scripts, which are the arguments provided to the `--script` option or `safe,intrusive`, in case of a default script scan. The string `version` is appended, if version detection was enabled. The arguments afterwards are tried to be interpreted as script categories. This is done via a short Lua function hard-coded into `nse_init.cc` called `Entry`. If you take a look into the `script.db` you'll see that the `Entry` lines inside it are Lua function calls with a table as argument. The arguments that didn't produce any filenames are then interpreted as file or directory names themselves. If this also fails, the script scan is aborted. In the next stage the found files are loaded as chunks, each with its own environment, having read but not write access to the global name space and saved inside two globally accessible Lua tables: `hosttests` and `porttests` depending on the type of script. Because scripts only get loaded once, values stored inside variables during a script's execution against one host or port can be accessed when the same script runs against another target. This can be used to save computation time when a script is run against multiple targets. See Example 9.3, “Using local variables to save data.” During this stage scripts are also are also provided with a default `runlevel` (1.0), if they don't specify one themselves and a check is performed whether they contain an `action` and a `description` field. Example 9.3. Using local variables to save data.
    id="persistent locals example"
    description="This sample script shows how data can be stored across \
    several invocations of a script against multiple targets"
    author="Stoiko Ivanov"
    categories = {"safe"}

    require "shortport"
    portrule = shortport.portnumber(80)
    -- we have to declare the variable in the script's global scope
    -- because if we declare it inside the action it would get redefined
    -- with each call to the action
    local filecontent = nil
    require "strbuf"
    action= function(host, port)
      if(filecontent == nil) then
        filecontent = strbuf.new()
        for line in io.lines("a_filename_we_want_to_read_from")
          filecontent = filecontent .. line
        end
      end
      --rest of the script doing something with the filecontent, we just
      --read
    end
### Matching of Scripts to Targets After the initialization is finished the `hostrules` and `portrules` are evaluated for each host in the current target group. At this check a list is built which contains the combinations of scripts and the hosts they will run against. It should be noted that the rules of all chosen scripts are checked against all hosts and their `open` and `open|filtered` ports. Therefore it is advisable to leave the rules as simple as possible and to do all the computation inside the `action`, as a script will only be executed if it is run against a specific target. After the check those script-target combinations get their own Lua-thread which is anchored in Lua's C-API registry to prevent their garbage collection. These `thread_records` are afterwards sorted by run level and all script-target combinations of one run level are stored in a list, in order to ensure that scripts with a higher run level are run after those with a lower one. ### Running Scripts Now to the actual script scanning, and the way NSE accomplishes parallelization. Lua, through its concept of coroutines offers collaborative multi-threading, which means that scripts can suspend themselves, at defined points, and let other coroutines execute. Since network I/O, especially waiting for responses from remote host, is the part of scripts which would consume most time with waiting, this is the point where scripts suspend themselves and let others execute. Each call to some of the functions of the Nsock wrapper causes the calling script to yield (pause). Once the request is processed by the Nsock library, the callback causes the script to be pushed from the waiting queue to the running queue, which will eventually let it resume its operation. ### Adding C Modules to Nselib This section tries to give a short walk-through to adding nselib modules written in C (or C++) to Nmap's build system, since this has shown to be sometimes tedious. Writing C modules is described at length in Programming in Lua, Second Edition Basically C modules consist of the functions they provide to Lua, which have to be of type lua_CFunction Additionally they have to contain a function which is used to actually open the module. By convention these function names are `luaopen_`modulename``. A good starting point for writing such modules is provided with `bit.c` and `pcre.c` inside the `nselib/` subdirectory of Nmap's source tree, which are two C modules already provided by the nselib. C modules basically are shared libraries which get loaded at runtime by Lua. The Unix build system uses `libtool` for compilation in a platform independent way. As long as the new module does not depend on foreign libraries, you should only need to add ``modulename`.so` to the `all` and `clean` targets in `Makefile.in` and copy and adapt the lines from `bit.so`. If your module does have dependencies you will most probably have to add checks for those libraries to `configure.ac` and put the dependencies inside the `libtool` commands in `Makefile.in`—here you should take a look at how `pcre.so` handles this. So much for the way it should work. Now for some pitfalls we've come across so far: These are results from building shared libraries in general and not really specific to nselib. Linking with static libraries (e.g. `libnbase`) sometimes leads to problems with exporting symbols on some platforms (in our case this was a x86_64-linux platform). To our knowledge no such problems occur when linking against already existing shared libraries. The Windows build system requires C module developers to create a MS Visual Studio Project file for their module (`<modulename>.vcproj`) inside the `nselib` subdirectory. On Windows you have to include the `liblua/` subdirectory as an additional include path as well as a library search path. In addition you have to tell the project to link against the `liblua.lib` static library provided with Nmap. Other properties of the project should be the same as for other nselib C modules (e.g. see `nse_bitlib.vcproj`). Afterwards you have to include the newly created project file in Nmap's Visual Studio solution file (`mswin32\nmap.sln`) and make sure that `nse_bitlib.vcproj` depends on your project, because it is there nselib-modules get copied to their final destinations and are included in Nmap.
Example Scripts |  | NSE Script License and Community Contributions
| [ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]