Welcome to the OpenTPX project!

Open Threat Partner Exchange (OpenTPX) is an open-source contribution that provides machine-readable threat intelligence, combining network security operations data with threat intelligence, analysis and scoring data in an optimized manner.

It is maintained by the OpenTPX community and licensed under Apache 2.0 License.

The technology has many benefits:

  • allows threat dictionary and meta-data to be conveyed only once between communicating systems.
  • allows transmission of hundreds of millions of network security observations based on that meta-data in a highly optimized mechanism, without requiring retransmission of the meta-data.
  • makes it possible for systems to convey all aspects of the network security and threat data in its most basic elements with minimal interpretation.
  • avoids complex or unwieldy language mappings that often confuse or block efforts to develop interoperable network security and threat intelligence systems.
  • allows OpenTPX content to be ingested by today's data processing frameworks and key-value data stores in a highly efficient manner.
  • allows complex queries to be combined with context.
  • provides a comprehensive threat-scoring framework that allows security analysts, threat researchers, network security operations and incident responders to make relevant threat mitigation decisions straight forward, while efficiently automating those decisions.

  • OpenTPX is an open-source JSON-based data model enabling optimized Internet and threat intelligence sharing.

    Latest News

    Date Item
    02/24/2016 FireEye examples added and new blog post "Ingesting FireEye Events to Threat Intelligence Systems in OpenTPX"
    10/20/2015 OpenTPX 2.2.0 released as open source

    OpenTPX 2.2.0 schema:
    { "$schema": "http://json-schema.org/draft-04/schema#", "title": "OpenTPX", "description": "An Open Threat Partner eXchange (OpenTPX) file", "definitions": { "suffixed_schema": { "type": "object", "patternProperties": { "_ipv4_i$": { "type": "integer", "minimum": 0, "maximum": 4294967295 }, "_ipv4_ui$": { "type": "integer", "minimum": 0, "maximum": 4294967295 }, "_ipv4_s$": { "type": "string", "format": "ipv4" }, "_cidrv4_s$": { "type": "string" }, "_ipv6_ll$": { "type": "integer", "minimum": 0 }, "_ipv6_s$": { "type": "string", "format": "ipv6" }, "_cidrv6_s$": { "type": "string" }, "_fqdn_s$": { "type": "string" }, "_asn_number_ui$": { "type": "integer", "minimum": 0 }, "_asn_s$": { "type": "string" }, "_md5_h$": { "type": "string", "pattern": "^[A-Fa-f0-9]{32}$" }, "_sha1_h$": { "type": "string", "pattern": "^[A-Fa-f0-9]{40}$" }, "_sha256_h$": { "type": "string", "pattern": "^[A-Fa-f0-9]{64}$" }, "_sha512_h$": { "type": "string", "pattern": "^[A-Fa-f0-9]{128}$" }, "_t$": { "type": "integer", "minimum": 0 }, "_s$": { "type": "string" }, "_i$": { "type": "integer" }, "_ui$": { "type": "integer" }, "_ll$": { "type": "integer" }, "_h$": { "type": "string", "pattern": "^[A-Fa-f0-9]+$" }, "_f$": { "type": "number" }, "_c_array$": { "type": "array", "minItems": 1, "items": { "type": "object" } }, "_c_map$": { "type": "object" }, "_s_array$": { "type": "array", "minItems": 1, "items": { "type": "string"} } }, "additionalProperties": false }, "observable": { "allOf": [{ "type": "object", "properties": { "observable_id_s": { "description": "The name of the observable", "type": "string" }, "criticality_i": { "description": "The threat observable’s relative criticality between 1 and 100", "type": "integer", "minimum": 1, "maximum": 100 }, "score_i": { "description": "Optional overridden threat score between 1 and 100.", "type": "integer", "minimum": 1, "maximum": 100 }, "score_24hr_decay_i": { "description": "Optional parameter that defines the percentage of the score decays over time if no new observation. A valid decay is between 0 and 100. A value of 0 switches off decay due to time.", "type": "integer", "minimum": 0, "maximum": 100 }, "score_calc_setting_s":{ "description": "Optional parameter that defines whether the score was calculated based on a manual or automatic calculation. Default: auto", "type": "string", "pattern": "^(auto|manual)$" }, "description_s": { "description": "A user displayable description of the observable", "type": "string" }, "classification_c_array": { "description": "An array of classification of this threat observable.", "type": "array", "minItems": 1, "items": { "allOf": [{ "type": "object", "properties": { "classification_id_s": { "description": "The name of the classification", "type": "string" }, "classification_family_s": { "description": "The descriptive family name for this classification", "type": "string" }, "score_i": { "description": "The criticality/score of the classification between 1 and 100 where a higher number is a higher risk classification", "type": "integer", "minimum": 1, "maximum": 100 } }, "additionalProperties": false, "required": ["classification_id_s"] }, { "$ref": "#/definitions/suffixed_schema" }] } }, "attribute_c_map": { "description": "An map of attributes associated with the observable that are common across all subjects", "$ref": "#/definitions/suffixed_schema" }, "summary_s": { "description": "A user displayable summary of the observable description", "type": "string" }, "notes_s": { "description": "A user defined set of notes that provide background to the description", "type": "string" }, "reference_s_array": { "description": "An array of string URL references to background information on the observable", "type": "array", "minItems": 1, "items": { "type": "string", "format": "uri" } } }, "required": ["observable_id_s", "description_s", "classification_c_array"] }, { "$ref": "#/definitions/suffixed_schema" }] }, "network": { "type": "object", "properties": { "asn_i": { "description": "The ID number of the ASN", "type": "integer" }, "as_owner_s": { "description": "The owner of the ASN", "type": "string" }, "asn_routers_ip_array": { "description": "The array of routers that make up this ASN", "type": "array", "minItems": 1, "items": { "type": "string", "pattern": ".*" } }, "asn_router_conns_ip_array": { "description": "The array of router interconnections in this ASN", "type": "array", "minItems": 1, "items": { "type": "string", "pattern": ".*" } }, "asn_cidr_announcements_c_array": { "description": "The array of CIDR announcements in this ASN", "type": "array", "minItems": 1, "items": { "$ref": "#/definitions/suffixed_schema" } }, "asn_downstream_i_array": { "description": "The array of downstream ASNs from this ASN", "type": "array", "minItems": 1, "items": { "type": "integer" } }, "asn_upstream_i_array": { "description": "The array of upstream ASNs from this ASN", "type": "array", "minItems": 1, "items": { "type": "integer" } }, "asn_community_c_array": { "description": "The array of communities within this ASN", "type": "array", "minItems": 1, "items": { "$ref": "#/definitions/suffixed_schema" } } } }, "collection": { "type": "object", "properties": { "name_id_s": { "description": "The name of the collection", "type": "string" }, "last_updated_t": { "description": "The UTC Epoch time of the last update to this collection", "type": "integer", "minimum": 0 }, "author_s": { "description": "A name associated with the last team, group, company or person making the change", "type": "string" }, "workspace_s": { "description": "A collaboration space this collection is associated with", "type": "string" }, "fqdn_ref_c_array": { "description": "An array of FQDN elements referenced by this collection", "type": "array", "minItems": 1, "items": { "$ref": "#/definitions/suffixed_schema" } }, "ip_ref_c_array": { "description": "An array of IP (v4 and v6) elements referenced by this collection", "type": "array", "minItems": 1, "items": { "$ref": "#/definitions/suffixed_schema" } }, "asn_ref_c_array": { "description": "An array of ASN elements referenced by this collection", "type": "array", "minItems": 1, "items": { "$ref": "#/definitions/suffixed_schema" } }, "cidr_ref_c_array": { "description": "An array of CIDR elements referenced by this collection", "type": "array", "minItems": 1, "items": { "$ref": "#/definitions/suffixed_schema" } }, "observable_ref_c_array": { "description": "An array of observables referenced by this collection", "type": "array", "minItems": 1, "items": { "$ref": "#/definitions/suffixed_schema" } }, "collection_c_array": { "description": "An array of children collections contained within this collection", "type": "array", "minItems": 1, "items": { "$ref": "#/definitions/collection" } } }, "required": ["name_id_s"] }, "element_observable": { "allOf": [{ "type": "object", "properties": { "score_i": { "description": "The element’s overridden score if not derived from scoring of the observables", "type": "integer", "minimum": 1, "maximum": 100 }, "score_24hr_decay_i": { "description": "The element’s overridden score decay if not derived from the observable’s decay parameter. 0 indicates this particular element’s score will not change due to time decay alone.", "type": "integer", "minimum": 0, "maximum": 100 }, "threat_observable_c_map": { "description": "A map of Threat Observables that are associated with the subject. The threat observable must already be defined in the observable dictionary to be referenced by this map.", "type": "object", "patternProperties": { ".+": { "allOf": [{ "type": "object", "properties": { "occurred_at_t": { "description": "The Epoch UTC timestamp when this particular threat observable was first observed associated with the subject", "type": "integer", "minimum": 0 }, "last_seen_t": { "description": "The Epoch UTC timestamp of the last update when this threat observable was observed associated with the subject", "type": "integer", "minimum": 0 }, "country_code_s": { "description": "The 2 or 3 digit country code associated with the threat observable", "type": "string", "minLength": 2, "maxLength": 3 }, "destination_fqdn_s": { "description": "The domain that a particular botnet or peer to peer communication threat was destined to", "type": "string" }, "description_s": { "description": "The description of the observable or element or collection", "type": "string" }, "url_s": { "description": "The description of the observable or element or collection", "type": "string", "format": "uri" }, "score_i": { "description": "The criticality/score of the classification between 1 and 100 where a higher number is a higher risk observable", "type": "integer", "minimum": 1, "maximum": 100 }, "classification_s": { "description": "The name of the classification", "type": "string" }, "filesize_i": { "description": "The size of a file used to convey some behavior", "type": "integer", "minimum": 0 }, "magic_s": { "description": "The description of the file", "type": "string" }, "mime_type_s": { "description": "The mime type of the file", "type": "string" }, "hash_md5_h": { "description": "The MD5 hash of a file", "type": "string" }, "hash_sha1_h": { "description": "The SHA1 hash of a file", "type": "string" }, "hash_sha256_h": { "description": "The SHA256 hash of a file", "type": "string" }, "hash _sha512_h": { "description": "The SHA512 hash of a file", "type": "string" }, "dns_request_c_array": { "description": "The list of DNS requests made", "type": "array", "items": { "$ref": "#/definitions/suffixed_schema" } }, "dns_response_c_array": { "description": "The list of DNS responses where each response is { Dns-record-type : Dns-value}", "type": "array", "items": { "$ref": "#/definitions/suffixed_schema" } }, "host_c_array": { "description": "The list of hosts in the PCAP", "type": "array", "items": { "$ref": "#/definitions/suffixed_schema" } }, "http_c_array": { "description": "The list of HTTP key/value pairs in the PCAP", "type": "array", "items": { "$ref": "#/definitions/suffixed_schema" } }, "smtp_c_array": { "description": "The list of SMTP key/value pairs in the PCAP", "type": "array", "items": { "$ref": "#/definitions/suffixed_schema" } }, "tcp_c_array": { "description": "The list of TCP key/value pairs in the PCAP", "type": "array", "items": { "$ref": "#/definitions/suffixed_schema" } }, "fqdn_c_array": { "description": "The list of SMTP key/value pairs in the PCAP", "type": "array", "items": { "$ref": "#/definitions/suffixed_schema" } }, "ssl_c_array": { "description": "The list of SSL key/value pairs in the PCAP", "type": "array", "items": { "$ref": "#/definitions/suffixed_schema" } }, "geoloc_lat_f": { "description": "The latitude of the observable if known", "type": "number" }, "geoloc_long_f": { "description": "The longitude of the observable if known", "type": "number" }, "dest_port_i": { "description": "A destination protocol port", "type": "integer" }, "dest_ipv4_s": { "description": "A destination IP v4 address as a string", "type": "string" }, "dest_ipv4_i": { "description": "A destination IP v4 address as an integer", "type": "integer" }, "src_port_i": { "description": "A source protocol port", "type": "integer" }, "src_ipv4_s": { "description": "A source IP v4 address as a string", "type": "string" }, "src_ipv4_i": { "description": "A source IP v4 address as an integer", "type": "integer" }, "size_i": { "description": "A size in bytes of a communication or entity", "type": "integer" }, "tlp_i": { "description": "The Traffic Light Protocol value. 0 – White, 1 – Green, 2 – Amber, 3 – Red", "type": "integer" }, "name_id_s": { "description": "The name of the country provided as part of a country code file", "type": "string" }, "country_code_i": { "description": "The country identifier as part of the country code file", "type": "integer", "minimum": 0 }, "iso_3_s": { "description": "The ISO 3 letter code for the country", "type": "string", "minLength": 3, "maxLength": 3 }, "iso_2_s": { "description": "The ISO 2 letter code for the country", "type": "string", "minLength": 2, "maxLength": 2 }, "region_code_i": { "description": "The regional code for the country code file", "type": "integer" }, "continent_code_i": { "description": "The continent code for the country code file", "type": "integer" }, "continent_code_s": { "description": "The continent name for the country code file", "type": "string" }, "naics_code_i": { "description": "The NAICS code", "type": "integer" }, "naics_code_s": { "description": "The NAICS code as a string", "type": "string" } }, "required": ["occurred_at_t"] }, { "$ref": "#/definitions/suffixed_schema" } ] } } } }, "required": ["threat_observable_c_map"] }, { "$ref": "#/definitions/suffixed_schema" }, { "oneOf": [ { "required": ["subject_ipv4_i"] }, { "required": ["subject_ipv4_ui"] }, { "required": ["subject_ipv4_s"] }, { "required": ["subject_ipv6_ui"] }, { "required": ["subject_ipv6_s"] }, { "required": ["subject_fqdn_s"] }, { "required": ["subject_cidrv4_s"] }, { "required": ["subject_cidrv6_s"] }, { "required": ["subject_asn_s"] }, { "required": ["subject_asn_ui"] }, { "required": ["subject_md5_h"] }, { "required": ["subject_sha1_h"] }, { "required": ["subject_sha256_h"] }, { "required": ["subject_sha512_h"] }, { "required": ["subject_registrykey_s"] }, { "required": ["subject_filename_s"] }, { "required": ["subject_filepath_s"] }, { "required": ["subject_mutex_s"] }, { "required": ["subject_actor_s"] }, { "required": ["subject_email_s"] } ] } ] } }, "type": "object", "properties": { "schema_version_s": { "description": "The provider’s version of their schema", "type": "string" }, "provider_s": { "description": "The provider’s company name", "type": "string" }, "source_observable_s": { "description": "The prefix associated with this threat list", "type": "string" }, "source_description_s": { "description": "A description of the source feed that provides background to the type of data, the types of information available to the user", "type": "string" }, "source_file_s": { "description": "The file containing the original feed information", "type": "string", "format": "uri" }, "score_i": { "description": "The score of the source feed accuracy. As assessment of the source feed’s accuracy between 1 and 100 where 100 is completely accurate", "type": "integer", "minimum": 1, "maximum": 100 }, "last_updated_t": { "description": "The Epoch UTC timestamp this file was last changed by the provider", "type": "integer", "minimum": 0 }, "distribution_time_t": { "description": "The Epoch UTC timestamp this file was distributed by the provider", "type": "integer", "minimum": 0 }, "list_name_s": { "description": "The threat feed list name", "type": "string" }, "observable_dictionary_c_array": { "description": "An array of observable definitions", "type": "array", "minItems": 0, "items": { "$ref": "#/definitions/observable" } }, "element_observable_c_array": { "description": "An array of Element Threat Observables", "type": "array", "minItems": 1, "items": { "$ref": "#/definitions/element_observable" } }, "collection_c_array": { "description": "An array of Collections", "type": "array", "minItems": 1, "items": { "$ref": "#/definitions/collection" } }, "asn_c_array": { "description": "An array of ASN network information", "type": "array", "minItems": 1, "items": { "$ref": "#/definitions/network" } }, "dictionary_file_manifest": { "description": "An array of filenames (fully qualified path) where the dictionary files are", "type": "array", "minItems": 1, "items": { "type": "string" } }, "observable_element_file_manifest": { "description": "An array of filenames (fully qualified path) where the element observable files are", "type": "array", "minItems": 1, "items": { "type": "string" } }, "collection_file_manifest": { "description": "An array of filenames (fully qualified path) where the collection files are", "type": "array", "minItems": 1, "items": { "type": "string" } }, "network_file_manifest": { "description": "An array of filenames (fully qualified path) where the network files are", "type": "array", "minItems": 1, "items": { "type": "string" } } }, "required": ["schema_version_s", "provider_s", "source_observable_s", "last_updated_t", "list_name_s"] }
    Get Started

    Current version: 2.2.0

    Docs

    Please have a look at the OpenTPX schema. You can also find our FAQ pages online.

    The following setup guides have been contributed by members of the OpenTPX Community for your use. Comments and questions on these documents should be submitted to our Mailinglist or by Github.

    For any questions on the content please contact our Mailinglist.

    Documents

    File  Date   Title  Size 
    openTPX-announcement.docx 10/20/2015  LookingGlass Introduces Open Threat Partner eXchange (OpenTPX) to Foster Enhanced Exchange of Network Security Intelligence 24 KB
    opentpx-2-2-0-spec.pdf 10/09/2015  Open Threat Partner Exchange (OpenTPX) Version 2.2.0 929 KB
    openTPX-introduction.pdf 10/09/2015  OpenTPX v2.2 2045 KB
    openTPX-faq.pdf 10/09/2015  OpenTPX FAQ 457 KB

    Downloads

    OpenTPX allows internet and threat intelligence to be shared in both small and large data sizes. For content that is greater than 1GByte then we recommend using the manifest concept, otherwise using a single file to contain all content may be appropriate.

    Single File Examples

    File  Description 
    tpx2-2-example-malware-report-nc.json An example that shows a malware report including IOCs, and a set of network entities that have been associated with that malware
    tpx2-2-example-malware-report-2-nc.json An example that shows a malware report including IOCs including meta-data without network associations
    tpx2-2-example-bgp-nc.json An example that shows internet intelligence and topology information sharing including BGP networks
    tpx2-2-example-collections-nc.json An example that shows segmentation concepts groups as collections of other network and domain entities
    tpx2-2-example-countrycodes.json An example that shows a collection defining the set of ISO-2 and ISO-3 country codes.
    tpx2-2-example-emailobservable.json An example that shows a data leak capture of email account information
    tpx2-2-example-ip-observables-nc.json An example that shows a set of threat observable dictionary definitions and their association to network entities
    tpx2-2-example-pcap-observables-nc.json An example that shows a set of threat observables associated with packet capture and behaviors of malware

    Manifest File Examples

    File  Description 
    tpx2-2-manifest-example-file-manifest-nc.json An example file manifest that acts an index to a set of other content
    tpx2-2-manifest-example-collections-nc.json An example of collections when contained in a set of files distributed as part of manifest
    tpx2-2-manifest-example-ddos-dictionary-nc.json An example of an observable dictionary for a DDoS threat observable
    tpx2-2-manifest-example-ddos-manifest.json An example of an observable association file to network entities.
    tpx2-2-manifest-example-ddos-observables-nc.json An example of an observable association file to network entities.
    tpx2-2-manifest-example-example-ip-dictionary-nc.json An example of an observable dictionary defining a threat that would be associated with IP network entities
    tpx2-2-manifest-example-ip-observables-nc.json An example of an observable association file association threats with IP network entities.
    tpx2-2-manifest-example-pcap-dictionary-nc.json An example that shows a set of threat observables in a dictionary for packet capture threats
    tpx2-2-manifest-example-pcap-observables-nc.json An example that shows the observable association to network entities of packet capture threats

    FireEye Examples

    File  Description 
    web-infection_ext.tpx.json Detailed web infection example. Contains added information on file changes, regkeys, Windows API calls, and mutexes.
    web-infection_concise.tpx.json Concise example of an infected website exploiting the browser.
    malware-object_ext_1.tpx.json Detailed malicious file example. Contains added information on file changes, regkeys, API calls, mutexes, and network traffic.
    malware-object_ext_2.tpx.json Another detailed malicious file example showing a different scenario. Contains additional malware callback information.
    malware-object_concise.tpx.json Concise example of a malicious file.
    malware-callback_ext.tpx.json Detailed example of a malware callback event. Contains information on the CnC server and it's network traffic.
    infection-match_ext.tpx.json Detailed example of an infected host. Contains additional callback information.
    infection-match_concise.tpx.json Concise example of an infected host on the network.
    domain-match_ext.tpx.json Detailed example of a malicious domain. Contains additional information on the malware as well as network traffic.
    domain-match_concise.tpx.json Concise example of a malicious domain.

     

    Please make sure to look at the README and INSTALL files in the distribution. If you have problems, look at the FAQ, which can also be found online.

    Community

    OpenTPX is a a collaborative effort of a worldwide community of volunteers. Here are some of the ways you can join the community and contribute.

    We maintain a mailing list. Anyone can join, but you must be a member of a list to post to it. We have a team blog, where members of the development team will occasionally post.

    You can also create a pull request in GitHub.

    Acknowledgement

    This technology includes software written by LookingGlass Cyber Solutions.

    Contact

    Send questions or comments to the OpenTPX community via this form.

    If you would like to contact LookingGlass team about OpenTPX technology please visit our web site at www.lookingglasscyber.com/about-us/contact-us.

    FAQ

    About
    Is a JSON based data model to convey all aspects of threat intelligence, threat analysis, threat mitigation and network security operation necessary for multiple security and threat intelligence use cases.
    Open(-source) Threat Partner eXchange.
    Version 2.2.0 is the current version. Prior versions of OpenTPX were not distributed as open source but helped the development team evolve and mature the specification prior to a broader consumption.
    No immediate plans beyond publishing the specification to the open source community.
    License, Use & Contributions
    Yes. Provided Apache 2.0 license rules are followed.
    Yes. Contributions are encouraged for the broader community to www.opentpx.org.
    Use Cases
    Machine to machine and human to machine exchange of any threat and Internet intelligence context worth sharing.
    Examples:
    • Automated threat intelligence feed indicating botnet infections
    • Human analyst report describing behaviors
    • BGP Topologies
    • Market segmentation of domains
    • Flow and packet captures associated with threat behaviors
    No. OpenTPX was designed primarily as a optimized mechanism for data exchange at large volume, high scale and high speed ingestion for a broader set of Internet intelligence and threat context. Aspects of data available in STIX (e.g. indicators) have direct mapping to OpenTPX.
    Any transport mechanism suitable to conveying JSON content including HTTP, SMTP, Syslog, SNMP, FTP etc.