Command Line Tools

Command-line Compiler Application

Included with the SDK is a command-line application that can be used to compile PCRE expressions or ANML files into an automaton. The application will compile a set of PCRE expressions or a single ANML "program" from a text file. It can also compile a single PCRE expression passed as a command line argument, optionally exporting the expression into an ANML file, using the -a option. When invoking the application, the last argument will be any of these. If compiling a single expression, make sure shell processing is turned off for the PCRE argument by enclosing it in quotes (double on Windows and single on Linux). A PCRE expression, whether on the command line or in a file must be structured as follows:

1 [<report code>: ] <regular expression>

Currently, the compiler application only supports a single PCRE expression per line. The PCRE API supports expressions on single or multiple lines with the AP_MOD_PCRE_EXTENDED modifier. For PCRE expressions in a file, if a report code is not present, the line number will be used as the report code. For a PCRE expression on the command line, if the report code is not present, it will default to 1. For ANML files, internally generated report codes are used to identify matches. However, if an ANML file is generated from a PCRE expression on the command line and a report code is present, all reporting elements will be assigned the given report code. Report codes can be converted to ANML "names," i.e. element IDs, as explained in the ANML Programmers Reference. A PCRE expression must be delimited with the standard '/' character. Any of the characters [imsAx] may appear after the trailing '/' to select the following modifiers:

  • 'i' for AP_MOD_CASELESS
  • 'm' for AP_MOD_MULTILINE
  • 's' for AP_MOD_PCRE_DOTALL
  • 'A' for AP_MOD_ANCHORED
  • 'x' for AP_MOD_PCRE_EXTENDED

The compiler is invoked as follows:

1 apcompile [options] <regex-or-ANML-file-name|regex>

The compiler operation can be modified by use of the options listed below.

  • --anml=<element-map-file>: Compile ANML Automata Network or precompile ANML Macro Definition file optionally specifying where to write the element map file.
  • -a|--generate-anml=<anml-file-name>: Generate an ANML file from a regular expression.
  • -E|--allow-states-lead-to-no-matching: Prevent states leading to no matching from being trimmed out.
  • -i|--include-macro=<anml-macro-file-name>: Include an external macro definition before compiling.
  • -l|--include-library=<anml-library-file-name>: Include an ANML library before compiling.
  • --anml-version=[0.8 | 1.0]: Specify the ANML schema version for reading ANML files. This option does not set the output schema when exporting.
  • -B|--backrefs: Enable support for PCRE back references.
  • -I|--ignore-unknown-mod: Ignore modifiers that are not part of PCRE specifications. If not set, these modifiers will cause an error.
  • -L|--lookaheads: Enable support for PCRE lookaheads and lookbehinds.
  • -N|--anycrlf: The default is changed so that \R matches only CR, LF, or CRLF.
  • -S|--strict: Enforce strict escape rules (like pcre strict). -SS adds strict rules on {,} quantifiers.
  • -U|--ignore-umod: Ignore PCRE's U modifier. If not set, these modifiers will cause an error.
  • -Od: Disable automaton minimization (enabled by default).
  • -O1|--best-quality: Enable high utilization placement at the expense of longer compile time (disabled by default).
  • -O4|--enable-first-match: Enable a more aggressive use of counters for quantifications (disabled by default). Ungreedy matching is preserved by this option but subsequent matches may be incorrect.
  • --disable-left-minimization: Disable left minimization on the automaton (enabled by default).
  • --disable-right-minimization: Disable right minimization on the automaton (enabled by default).
  • --merge-expression: Enable a more agressive optimization for single expression (disabled by default).
  • --merge-all-expressions: Enable a more aggressive minimization of the STE usage at the expense of not being able to determine which pattern caused the match (disabled by default).
  • -f|--output=<automaton-file-name>: Write the output to <automaton-file-name>. If no file is given stdout is assumed.
  • -K|--pwr-ctrl: Enable power management for all blocks.
  • -MT: Use multi-threading when possible.
  • -W|--warn: Treat errors as warnings and discard bad expressions.
  • -X|--exclude-route: Compile an output automaton but don't start placement and route.
  • -h|--help: Display this message.
  • -v|-vv|-vvv: Verbose, very-verbose, or very-very-verbose. This is used to see more detailed status or failure information.
  • -V|--version: Print the version.
  • --disable-counters: Disable the use of counters for quantifications.
  • --log=<filename>: Enables logging to filename.
  • --clearlog: Truncates the log file passed in –log, default is to append.

The errors returned by the compiler are listed below.

  • AP_ERR_CANNOT_FIND_SCHEMA_FILE if the schema referenced in the ANML file cannot be found
  • AP_ERR_DEVICE_FAMILY if the device family is unknown
  • AP_ERR_DUPLICATE_ACTIVATION if an element is activated more than once by the same element
  • AP_ERR_DUPLICATE_ID if any of the following conditions exist:
    • a duplicate ID is found
    • different external ANML macro definition files are referenced that each contain a macro with the same ID
  • AP_ERR_DUPLICATE_MACRO_PARAMETER if a duplicate macro parameter is defined
  • AP_ERR_DUPLICATE_SUBSTITUTION if a duplicate parameter substitution is found
  • AP_ERR_EXPR_EMPTY if the compilation resulted in an empty automaton
  • AP_ERR_INVALID_ACTIVATION if there is an inconsistent usage of a port – for example an external element activates an output port or an input port is specified to activate an element in a macro reference
  • AP_ERR_INVALID_ARGUMENT if any of the following conditions exist:
    • a Counter element's target is not in the range of 1-2048
    • the report code for a reporting element is not a number or is not in the range of 0-0x7fffffff
    • a recursive macro reference, that is, a macro reference that uses the macro definition in which the macro reference is contained
  • AP_ERR_INVALID_ELEMENT if any of the following conditions exist:
    • a macro definition has an empty body-binary
    • an element that is connected to a report alias is not a reporting element
  • AP_ERR_INVALID_FILE_FORMAT if an externally referenced macro definition file contains an error or does not contain a macro definition
  • AP_ERR_INVALID_ID if any of the following conditions exist:
    • an undefined ID is referenced – Note: a missing ID is an ANML syntax error (AP_ERR_ANML_SYNTAX)
    • an input port is connected directly to an output port
  • AP_ERR_INVALID_MACRO_PARAMETER if any of the following conditions exist:
    • a macro parameter is declared but unused
    • a macro parameter has an empty default value
  • AP_ERR_INVALID_PORT if a macro input, output or report port is undefined
  • AP_ERR_READ_FAILURE if an automata network or macro definition file cannot be read
  • AP_ERR_ANML_SYNTAX if the ANML file does not conform to the ANML schema
  • AP_ERR_SCHEMA_UNSUPPORTED if the schema referenced in the ANML file is not supported
  • AP_ERR_SYNTAX if an STE element contains an invalid symbol-set
  • AP_WARN_INCONSISTENT_REPORT_CODES if explicit report codes are not used consistently for match reporting elements

NOTE: ANML files can contain references to external macro definition files. When a containing ANML file, either an automata network file or a macro definition is compiled, each referenced macro definition file is compiled when it is encountered. If there is an error in the external macro definition file, the compiler will emit the AP_ERR_INVALID_FILE_FORMAT error and the line number where the file was referenced. In order to debug such a situation, use the line number to determine which external macro definition files is at fault, then compile that file alone, using the -A|--anml option. This will pinpoint the error in the offending file. If there are nested macro definition files, this process should be repeated for each level of nesting.

Command-line Emulator Application

Included with the SDK is a command-line application that can be used to emulate a hardware search. The emulator application takes a compiled automaton and searches against input data from either a text file, or from a single line of text passed as a command line argument. When invoking the application, the last argument will be either of these. The input data format supports binary input by use of escape sequences.

The emulator is invoked as follows:

1 apemulate [options] <automaton-file-name> [automaton-file-name] <data-file-name|data>

The emulator operation can be modified by use of the options listed below.

  • -m|--element_map=<element-map-file-name>: Display element ids in match results when element map file <element-map-file-name> is given.
  • -f|--output=<match-file-name>: Write the match result to <match-file-name>. If no file is given stdout is assumed.
  • -d|--device=<device-count>: Specify the number of devices to be used in Emulator. The default is 1.
  • -l|--literal: Process input string literally. The default is to un-escape input according to the below rules. '\' can also be used as a continuation character when it is followed by a newline character. If '\' is the last character, it will be ignored.
        Escape sequence    Character
             \a            0x07 alarm BEL
             \b            0x08 backspace
             \e            0x1b escape
             \f            0x0c formfeed
             \n            0x0a newline
             \r            0x0d carriage return
             \t            0x09 horizontal tab
             \v            0x0b vertical tab
             \"            0x22 double quotes
             \'            0x27 single quote
             \\            0x5c back slash
             \?            0x3f question mark
             \ooo          octal value
             \xdd          hexadecimal value
  • -h|--help: Display this message.
  • -a|--all: Print out detailed match results.
  • -v|-vv: Verbose or very-verbose.
  • -V|--version: Print the version.

Command-line Automaton Administration Application

Included with the SDK is a command-line application that can be used for general automaton administration. The application allows the use to either post-process or query an automaton after it has been created with the compiler.

The administrator is invoked as follows:

1 apadmin [options] <automaton-file-name|anml-element-map-file-name>

The apadmin operations are listed below.

  • -L|--list: List the properties of an automaton.
  • -P|--schema-path: Display the schema path.
  • -e|--expressions: List the expression IDs.
  • -f|--output=<automaton-file-name>: Specify an output file – used with the –run and –extract options.
  • -g|--graphviz=<graphviz-file-name>: Save a graphviz diagram of the automaton. This can viewed using dotty.
  • -y|--yed=<yed-file-name>: Save a graphml diagram of the automaton. This can viewed using yEd.
  • -Y|--yed-ports: Combined with the -y|–yed option this flag enables the use of graphml ports.
  • -r|--run=[place-all|place-failed|route-all|route-failed|merge]: Abbreviations: place for place-failed, route for route-failed.
  • -n|--subgraph=<integer-index>: Used with the –run option, limits routing or placement to a specific subgraph id.
  • -O1: Optimization setting for high utilization placement at the expense of longer compile time.
  • -c|--check=[load|route|place]: Check load, route, or placement feasibility.
  • -K|--pwr-ctrl: Disable power management for all blocks.
  • -t|--timing-target=<1|2|3>: Used with –run[place-all|place-failed] option. Sets the timing symbol clock multiplier. Clock=timing-target*7.45ns.
  • -X|--extract: Extract a subgraph from the input automaton. Only valid with -n|–subgraph option.
  • -A|--anml-element-map: Indicates an ANML element map file is specified instead of an automaton.
  • -m|--element-map=<text-file-name>: Save a text version of the ANML element map file.
  • -MT: Use multi-threading when possible.
  • -h|--help: Display this message.
  • -v|-vv: Verbose or very-verbose
  • -V|--version: Print the version.

Examples:

Create a graph of the automaton "my_automaton.fsm"

1 apadmin -y my_automaton.graphml my_automton.fsm

List the properties of the automaton "my_automaton.fsm"

1 apadmin -L automaton.fsm

Create a human-readable text file from the binary ANML element map "my_anml_map.map"

1 apadmin -Am my_anml_map.txt my_anml_map.map

Create a human-readable text file from and list the properties of the binary ANML element map "my_anml_map.map"

1 apadmin -ALm my_anml_map.txt my_anml_map.map

Command-line File Scan Application

Included with the SDK is a command-line application that can be used to scan input files and execute them on automata on the AP hardware. The application uses previously loaded automata (under a specific sharename), or can load one or more fsm files onto the hardware. It iterates through all files in a directory, and for each file it sends its contents as input to the automata, reporting match data at the end. The application can spawn multiple threads and divide the files between the threads. The application can also fork more than one process where each process scans all the files in the scan directory.

To run the application, use the following command:

1 apfilescan [options] sharename /path/to/directory/to/scan

The application can also generate a prediction file using the emulator for all expected matches, and can compare the actual matches from the hardware to the prediction file, reporting only mismatches. To use this modes, use the following commands:

1 apfilescan -f /path/to/fsm [-f /path/to/fsm ...] -x /path/to/prediction/file sharename /path/to/directory/to/scan
2 apfilescan [options excluding fx] -X sharename /path/to/prediction/file

The file scan options are listed below.

  • -d|--device=</path/to/device/node>: Specify the device to use. Defaults to the first device listed at /proc/ap/devices
  • -m|--element_map=<element-map-file-name>: Display element ids in match results when element map file <element-map-file-name> is given.
  • -n|--bytes-to-read=<integer>: Number of bytes to read from each file.
  • -f|--fsm=</path/to/fsm/file>: FSM to load. Multiple fsms may be loaded by using this flag multiple times.
  • -p|--processes: Number of processes to fork.
  • -t|--threads=<integer>: Number of threads to read file.
  • -M|--files-to-read=<integer>: Number of files that can be selected for flow scan operations.
  • -N|--files-to-open=<integer>: Number of files to open flows against.
  • -x|--create-predict=/path/to/write/prediction/file: Use the emulator to predict expected match events and save to a file. The prediction file and -X will only print matches that are not predicted.
  • -X|--predict: Only print matches that are not predicted.
  • -W|--nowait: Don't wait and process matches after each scan. Default is to wait.
  • -E|--noeod: Disable match on end-of-data. Default is enabled.
  • -h|--help: Display help information.
  • -v|--verbose: Display matches and other info.
  • -V|--version: Print the version and exit.

Command-line File Load Application

Included with the SDK is a command-line application that is used to load automata files so that they can later be executed on AP hardware with an AP application such as the SDK's apfilescan command line application. The application loads the automaton file and associates it with a specific sharename.

To run the application, use the following command:

1 apload [options] sharename <filepath|existing-share> [<filepath|existing-share>...]

Where filepath can be an automaton or a list of automaton paths. The alternative is to specify an existing share name to associate with the file.
Do the following to see existing shares:

1 cat /proc/ap/devices/<device name>/shares

The file load options are listed below.

  • -d|--device=</path/to/device/node>: Specify the device to use. Defaults to the first device listed at /proc/ap/devices
  • -i|--id=<load-region-id>: Specify the load region id. Default is 0.
  • -h|--help: Display this message.
  • -v|--verbose: Verbose.
  • -V|--version: Print the version.

Command-line File Unoad Application

Included with the SDK is a command-line application that is used to unload automata files after they have been executed on AP hardware with an AP application such as the SDK's apfilescan command line application. The application unloads the automaton file and frees it's associated sharename.

To run the application, use the following command:

1 apunload [options] sharename

Pass [] as sharename to unload all.

The file load options are listed below.

  • -d|--device=</path/to/device/node>: Specify the device to use. Defaults to the first device listed at /proc/ap/devices
  • -h|--help: Display this message.
  • -v|--verbose: Verbose.
  • -V|--version: Print the version.

Command-line SDK Registration Application

Included with the SDK is a command-line application that can be used to activate and deactivate the SDK. During activation, a license for the SDK is issued. During deactivation, the license is revoked. These two steps are required when installing and uninstalling to ensure we track activated products. All SDK users are issued a product key. The AP developers portal is the one and only way users are issued keys. The key is locked to a subscription level and SDK version.revision (not build), for example 1.4, basic evaluation. The process of activation and deactivation normally requires a network connection. The license issuing server is a cloud based service provided by Nalpeirion. As an alternative, a method for offline activation is provided for systems that do not have a direct internet connection. This requires another machine with internet access and a means to transport (typically via usb stick) the activation/deactivation request between machines.

The registration application is invoked as follows:
apregister [options] <product_key>

The apregister operations are listed below.

  • --autocfg=<PAC URL>: Set the URL of the Proxy Auto Config file.
  • --proxy=<url[:port]>: Set the proxy url and port.
  • --proxy-user=<username>: Set the username if the proxy requires authentication.
  • --proxy-passwd=<password>: Set the password if the proxy requires authentication.
  • --proxy-settings: Load proxy settings from configuration file (Linux defaults to /etc/apsdk/proxy.conf, Windows defaults to install settings) overriding command line settings.
  • --activation-request=<fileName>: Generate off-line activation request, sets the file name for activation request.
  • --import-license=<fileName>: Import license certificate from file, sets the file name for import.
  • --deactivation-request=<fileName>: Generate off-line deactivation request, sets the file name for deactivation request.
  • -L|--log: Enable syslogging (used in cron jobs).
  • -W|--wpad: Locate proxy auto config file automatically through WPAD.
  • -N|--netcheck: Verify the network connection.
  • -F|--refresh: Refresh license file.
  • -D|--dump: Print the license info for this machine.
  • -R|--revoke: Revoke the license for this machine.
  • -v|--verbose: Print error messages, default is silent.
  • -V|--version: Print the version.
  • -h|--help: Print this message.
  • -d: Exit with zero code if license is active (scripting helper).
  • -s: Exit with zero code if license matches subscription (scripting helper).

On-line SDK Registration

We support the following proxy specifications:

  1. Manual proxy selection --proxy.
  2. Automatic proxy configuration from a Proxy Auto-Configuration script (PAC) --autocfg. The equivalent internet explorer connection setting is "Use automatic configuration script."
  3. Web Proxy Auto-Detect (WPAD) --wpad. This is a general method of retrieving the PAC script. The equivalent internet explorer connection setting is "Automatically detect settings."

Both 2 and 3 end up using a PAC script. Despite its name, a PAC script may specify no proxy, i.e. direct connection, or it may specify failover proxies, with possible failover to direct connection. The apregister application will try all possible proxies in the order specified if failover is required. If a user specifies both --wpad and --autocfg then the WPAD is used. If the user specifies --proxy with either --wpad or --autocfg then the manual proxy is added to the list of proxy connections that will be tried. This follows the way internet explorer resolves proxies. We only support WPAD via DNS, i.e. DHCP and netbios are not supported (chrome and firefox have the same restriction). When installing on windows, the installer MSI gets the initial settings from the internet explorer settings. The exception is --wpad, since it is not clear where IE stores this flag in the windows registry. For any proxy configuration a username and password may be required. This is true for manual proxy, PAC, and WPAD.

To activate online:

  • Register the product: apregister [proxy setting] <productkey>

To deactivate online:

  • Unregister the product: apregister [proxy setting] --revoke

Off-line SDK Registration

In order to use apregister for off-line registration, you must have root privileges for Linux or run as an administrator for Windows, otherwise the registration will fail. If an unsuccessful registration has occurred in Widows, you will need to remove the files from the license folder under the SDK installation directory and generate a fresh activation request as administrator.

To activate offline:

  1. Create an activation request: apregister --activation-request=<activation_request_filename> <product-key>
  2. Copy the activation request file to a usb stick and go to a machine with internet access.
  3. Using a web browser, got to the off-line activation portal to activate your license.
  4. Copy the contents of the activation request file into the "Activation Certificate" field, then click "Activate."
  5. If successful, a "License" field will appear. Copy the contents to a file which will become the license certificate file.
  6. Copy the license certificate file to a usb stick and return to the machine with the SDK.
  7. Import the license certificate : apregister --import-license=<license_certificate_filename> <product-key>

To deactivate offline:

  1. Create a deactivation request: apregister --deactivation-request=<deactivation_request_filename>
  2. Copy the deactivation request file to a usb stick and go to a machine with internet access.
  3. Using a web browser, got to the off-line activation portal to deactivate your license.
  4. Copy the contents of the deactivation request file into the "Deactivation Certificate" field, then click "Deactivate."
  5. If successful, the license has been deactivated.