# Accessing a Neptune graph with Gremlin
/binaries/apache-maven--bin.tar.gz
sudo tar -xzf apache-maven--bin.tar.gz -C /opt/
sudo ln -sf /opt/apache-maven- /opt/maven
echo 'export MAVEN_HOME=/opt/maven' >> ~/.bashrc
echo 'export PATH=$MAVEN_HOME/bin:$PATH' >> ~/.bashrc
source ~/.bashrc
```
1. **Install Java.** The Gremlin libraries need Java 8 or 11. You can install Java 11 as follows:
+ If you're using [Amazon Linux 2 (AL2)](https://aws.amazon.com/amazon-linux-2):
```
sudo amazon-linux-extras install java-openjdk11
```
+ If you're using [Amazon Linux 2023 (AL2023)](https://docs.aws.amazon.com/linux/al2023/ug/what-is-amazon-linux.html):
```
sudo yum install java-11-amazon-corretto-devel
```
+ For other distributions, use whichever of the following is appropriate:
```
sudo yum install java-11-openjdk-devel
```
or:
```
sudo apt-get install openjdk-11-jdk
```
1. **Set Java 11 as the default runtime on your EC2 instance:** Enter the following to set Java 8 as the default runtime on your EC2 instance:
```
sudo /usr/sbin/alternatives --config java
```
When prompted, enter the number for Java 11.
1. **Create a new directory named `gremlinjava`:**
```
mkdir gremlinjava
cd gremlinjava
```
1. In the `gremlinjava` directory, create a `pom.xml` file, and then open it in a text editor:
```
nano pom.xml
```
1. Copy the following into the `pom.xml` file and save it:
```
UTF-8
4.0.0
com.amazonaws
GremlinExample
jar
1.0-SNAPSHOT
GremlinExample
https://maven.apache.org
org.apache.tinkerpop
gremlin-driver
3.7.2
org.slf4j
slf4j-jdk14
1.7.25
org.apache.maven.plugins
maven-compiler-plugin
2.5.1
11
11
org.codehaus.mojo
exec-maven-plugin
1.3
java
-classpath
com.amazonaws.App
com.amazonaws.App
1.11
-1
```
**Note**
If you are modifying an existing Maven project, the required dependency is highlighted in the preceding code.
1. Create subdirectories for the example source code (`src/main/java/com/amazonaws/`) by typing the following at the command line:
```
mkdir -p src/main/java/com/amazonaws/
```
1. In the `src/main/java/com/amazonaws/` directory, create a file named `App.java`, and then open it in a text editor.
```
nano src/main/java/com/amazonaws/App.java
```
1. Copy the following into the `App.java` file. Replace *your-neptune-endpoint* with the address of your Neptune DB instance. Do *not* include the `https://` prefix in the `addContactPoint` method.
**Note**
For information about finding the hostname of your Neptune DB instance, see [Connecting to Amazon Neptune Endpoints](feature-overview-endpoints.md).
```
package com.amazonaws;
import org.apache.tinkerpop.gremlin.driver.Cluster;
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversal;
import static org.apache.tinkerpop.gremlin.process.traversal.AnonymousTraversalSource.traversal;
import org.apache.tinkerpop.gremlin.driver.remote.DriverRemoteConnection;
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__;
import org.apache.tinkerpop.gremlin.structure.T;
public class App
{
public static void main( String[] args )
{
Cluster.Builder builder = Cluster.build();
builder.addContactPoint("your-neptune-endpoint");
builder.port(8182);
builder.enableSsl(true);
Cluster cluster = builder.create();
GraphTraversalSource g = traversal().withRemote(DriverRemoteConnection.using(cluster));
// Add a vertex.
// Note that a Gremlin terminal step, e.g. iterate(), is required to make a request to the remote server.
// The full list of Gremlin terminal steps is at https://tinkerpop.apache.org/docs/current/reference/#terminal-steps
g.addV("Person").property("Name", "Justin").iterate();
// Add a vertex with a user-supplied ID.
g.addV("Custom Label").property(T.id, "CustomId1").property("name", "Custom id vertex 1").iterate();
g.addV("Custom Label").property(T.id, "CustomId2").property("name", "Custom id vertex 2").iterate();
g.addE("Edge Label").from(__.V("CustomId1")).to(__.V("CustomId2")).iterate();
// This gets the vertices, only.
GraphTraversal t = g.V().limit(3).elementMap();
t.forEachRemaining(
e -> System.out.println(t.toList())
);
cluster.close();
}
}
```
For help connecting to Neptune with SSL/TLS (which is required), see [SSL/TLS configuration](#access-graph-gremlin-java-ssl).
1. Compile and run the sample using the following Maven command:
```
mvn compile exec:exec
```
The preceding example returns a map of the key and values of each property for the first two vertexes in the graph by using the `g.V().limit(3).elementMap()` traversal. To query for something else, replace it with another Gremlin traversal with one of the appropriate ending methods.
**Note**
The final part of the Gremlin query, `.toList()`, is required to submit the traversal to the server for evaluation. If you don't include that method or another equivalent method, the query is not submitted to the Neptune DB instance.
You also must append an appropriate ending when you add a vertex or edge, such as when you use the `addV( )` step.
The following methods submit the query to the Neptune DB instance:
+ `toList()`
+ `toSet()`
+ `next()`
+ `nextTraverser()`
+ `iterate()`
## SSL/TLS configuration for Gremlin Java client
org.apache.tinkerpop
gremlin-driver
${gremlin.version}
com.amazonaws
amazon-neptune-sigv4-signer
${sig4.signer.version}
com.evanlennick
retry4j
0.15.0
```
Here is the sample code:
**Important**
The `CallExecutor` from Retry4j may not be thread-safe. Consider having each thread use its own `CallExecutor` instance, or use a different retrying library.
**Note**
The following example has been updated to include the use of requestInterceptor(). This was added in TinkerPop 3.6.6. Prior to TinkerPop version 3.6.6, the code example used handshakeInterceptor(), which was deprecated with that release.
```
public static void main(String args[]) {
boolean useIam = true;
// Create Gremlin cluster and traversal source
Cluster.Builder builder = Cluster.build()
.addContactPoint(System.getenv("neptuneEndpoint"))
.port(Integer.parseInt(System.getenv("neptunePort")))
.enableSsl(true)
.minConnectionPoolSize(1)
.maxConnectionPoolSize(1)
.serializer(Serializers.GRAPHBINARY_V1D0)
.reconnectInterval(2000);
if (useIam) {
builder.requestInterceptor( r -> {
try {
NeptuneNettyHttpSigV4Signer sigV4Signer =
new NeptuneNettyHttpSigV4Signer("(your region)", new DefaultAWSCredentialsProviderChain());
sigV4Signer.signRequest(r);
} catch (NeptuneSigV4SignerException e) {
throw new RuntimeException("Exception occurred while signing the request", e);
}
return r;
});
}
Cluster cluster = builder.create();
GraphTraversalSource g = AnonymousTraversalSource
.traversal()
.withRemote(DriverRemoteConnection.using(cluster));
// Configure retries
// NOTE: This example uses a fixed backoff for simplicity.
// In a production application consider use of exponential backoff with jitter.
RetryConfig retryConfig = new RetryConfigBuilder()
.retryOnCustomExceptionLogic(getRetryLogic())
.withDelayBetweenTries(1000, ChronoUnit.MILLIS)
.withMaxNumberOfTries(5)
.withFixedBackoff()
.build();
@SuppressWarnings("unchecked")
CallExecutor retryExecutor = new CallExecutorBuilder()
.config(retryConfig)
.build();
// Do lots of queries
for (int i = 0; i < 100; i++){
String id = String.valueOf(i);
@SuppressWarnings("unchecked")
Callable query = () -> g.mergeV(Map.of(T.id, id))
.option(Merge.onCreate, Map.of(T.label, "Person"))
.id().next();
// Retry query
// If there are connection failures, the Java Gremlin client will automatically
// attempt to reconnect in the background, so all we have to do is wait and retry.
Status status = retryExecutor.execute(query);
System.out.println(status.getResult().toString());
}
cluster.close();
}
private static Function getRetryLogic() {
return e -> {
Class exceptionClass = e.getClass();
StringWriter stringWriter = new StringWriter();
String message = stringWriter.toString();
if (RemoteConnectionException.class.isAssignableFrom(exceptionClass)){
System.out.println("Retrying because RemoteConnectionException");
return true;
}
// Check for connection issues
if (message.contains("Timed out while waiting for an available host") ||
message.contains("Timed-out") && message.contains("waiting for connection on Host") ||
message.contains("Connection to server is no longer active") ||
message.contains("Connection reset by peer") ||
message.contains("SSLEngine closed already") ||
message.contains("Pool is shutdown") ||
message.contains("ExtendedClosedChannelException") ||
message.contains("Broken pipe") ||
message.contains(System.getenv("neptuneEndpoint")))
{
System.out.println("Retrying because connection issue");
return true;
};
// Concurrent writes can sometimes trigger a ConcurrentModificationException.
// In these circumstances you may want to backoff and retry.
if (message.contains("ConcurrentModificationException")) {
System.out.println("Retrying because ConcurrentModificationException");
return true;
}
// If the primary fails over to a new instance, existing connections to the old primary will
// throw a ReadOnlyViolationException. You may want to back and retry.
if (message.contains("ReadOnlyViolationException")) {
System.out.println("Retrying because ReadOnlyViolationException");
return true;
}
System.out.println("Not a retriable error");
return false;
};
}
```
# Using Python to connect to a Neptune DB instance
(3).ToList();
foreach(var item in output) {
Console.WriteLine(item);
}
}
catch (Exception e)
{
Console.WriteLine("{0}", e);
}
}
}
}
```
1. Enter the following command to run the sample:
```
dotnet run
```
The Gremlin query at the end of this example returns the count of a single vertex for testing purposes. It is then printed to the console.
**Note**
The final part of the Gremlin query, `Next()`, is required to submit the traversal to the server for evaluation. If you don't include that method or another equivalent method, the query is not submitted to the Neptune DB instance.
The following methods submit the query to the Neptune DB instance:
+ `ToList()`
+ `ToSet()`
+ `Next()`
+ `NextTraverser()`
+ `Iterate()`
Use `Next()` if you need the query results to be serialized and returned, or `Iterate()` if you don't.
The preceding example returns a list by using the `g.V().Limit(3).ToList()` traversal. To query for something else, replace it with another Gremlin traversal with one of the appropriate ending methods.
## IAM authentication
).out().out().out().out().out().out().out().out().out().out().limit(1).path()
```
**Important**
The Neptune query engine does not do this automatically.
Breadth-first (`BFS`) is the default execution strategy, and is similar to TinkerGraph in most cases. However, there are certain cases where depth-first (`DFS`) strategies are preferable.
**BFS (Default)**
Breadth-first (BFS) is the default execution strategy for the `repeat()` operator.
```
g.V("3").repeat(out()).times(10).limit(1).path()
```
The Neptune engine fully explores the first nine-hop frontiers before finding a solution ten hops out. This is effective in many cases, such as a shortest-path query.
However, for the preceding example, the traversal would be much faster using the depth-first (`DFS`) mode for the `repeat()` operator.
**DFS**
The following query uses the depth-first (`DFS`) mode for the `repeat()` operator.
```
g.withSideEffect("Neptune#repeatMode", "DFS").V("3").repeat(out()).times(10).limit(1)
```
This follows each individual solution out to the maximum depth before exploring the next solution.
# Gremlin noReordering query hint
("title").Limit(1000);
while (traversal.HasNext())
{
var batch = traversal.Next(batchSize);
batchNum++;
foreach (var title in batch)
{
Console.WriteLine($" {title}");
}
// Do other intermediary work here between batch calls
Console.WriteLine($"Batch {batchNum} processing complete\n");
}
```
**Example Node.js**
```
// The Node.js driver does not support next(n), so batches are accumulated manually.
const g = traversal().withRemote(connection);
const batchSize = 10;
let batchNum = 0;
const t = g.V().hasLabel('movie').values('title').limit(1000);
while (true) {
const batch = [];
for (let i = 0; i < batchSize; i++) {
const result = await t.next();
if (result.done) break;
batch.push(result.value);
}
if (batch.length === 0) break;
batchNum++;
for (const title of batch) {
console.log(` ${title}`);
}
// Do other intermediary work here between batch calls
console.log(`Batch ${batchNum} processing complete\n`);
}
```
## Eager vs. incremental consumption
<~label> <~>) .]
```
## Gremlin Edge Statements
) .]
```
## Gremlin Property Statements
"John" <~>) .]
```
Property statements differ from others in that their object is a primitive value (a `string`, `date`, `byte`, `short`, `int`, `long`, `float`, or `double`). Their object is not a resource identifier that could be used as the subject of another assertion.
For multi-properties, each individual property value in the set receives its own statement.
```
g.V("v1").property(set, "phone", "956-424-2563").property(set, "phone", "956-354-3692 (tel:9563543692)")
```
This results in the following.
```
StatementEvent[Added( "956-424-2563" <~>) .]
StatementEvent[Added( "956-354-3692" <~>) .]
```
Edge properties are handled similarly to vertex properties, but use the edge identifier in the (S) position. For example, adding a property to an edge:
```
g.E("e1").property("weight", 0.8)
```
This results in the following statement being added to the graph.
```
StatementEvent[Added( 0.8 <~>) .]
```
# How Neptune processes Gremlin queries using statement indexes
, <~label>, ?, ?)
Known positions: SP
Lookup positions: OG
Index: SPOG
Key range: :<~label>:*
```
**Question: What are the 'knows' out-edges of vertex `v1`?**
```
Gremlin code: g.V('v1').out('knows')
Pattern: (, , ?, ?)
Known positions: SP
Lookup positions: OG
Index: SPOG
Key range: ::*
```
**Question: Which vertices have a `Person` vertex label?**
```
Gremlin code: g.V().hasLabel('Person')
Pattern: (?, <~label>, , <~>)
Known positions: POG
Lookup positions: S
Index: POGS
Key range: <~label>::<~>:*
```
**Question: What are the from/to vertices of a given edge `e1`?**
```
Gremlin code: g.E('e1').bothV()
Pattern: (?, ?, ?, )
Known positions: G
Lookup positions: SPO
Index: GPSO
Key range: :*
```
One statement index that Neptune does **not** have is a reverse traversal OSGP index. This index could be used to gather all incoming edges across all edge labels, as in the following example.
**Question: What are the incoming adjacent vertices `v1`?**
```
Gremlin code: g.V('v1').in()
Pattern: (?, ?, , ?)
Known positions: O
Lookup positions: SPG
Index: OSGP // <-- Index does not exist
```
# How Gremlin queries are processed in Neptune
, , ?2, ?) X Pattern(?2, , ?3, ?)
The pattern produces a three-column relation (?1, ?2, ?3) like this:
?1 ?2 ?3
================
v1 v2 v3
v1 v2 v4
v1 v5 v6
```
By sharing the `?2` variable across the two patterns (at the O position in the first pattern and the S position of the second pattern), you create a join from the first hop neighbors to the second hop neighbors. Each Neptune solution has bindings for the three named variables, which can be used to re-create a [TinkerPop Traverser](http://tinkerpop.apache.org/docs/current/reference/#_the_traverser) (including path information).
```
```
The first step in Gremlin query processing is to parse the query into a TinkerPop [Traversal](http://tinkerpop.apache.org/docs/current/reference/#traversal) object, composed of a series of TinkerPop [steps](http://tinkerpop.apache.org/docs/current/reference/#graph-traversal-steps). These steps, which are part of the open-source [Apache TinkerPop project](http://tinkerpop.apache.org/), are both the logical and physical operators that compose a Gremlin traversal in the reference implementation. They are both used to represent the model of the query. They are executable operators that can produce solutions according to the semantics of the operator that they represent. For example, `.V()` is both represented and executed by the TinkerPop [GraphStep](http://tinkerpop.apache.org/docs/current/reference/#graph-step).
Because these off-the-shelf TinkerPop steps are executable, such a TinkerPop Traversal can execute any Gremlin query and produce the correct answer. However, when executed against a large graph, TinkerPop steps can sometimes be very inefficient and slow. Instead of using them, Neptune tries to convert the traversal into a declarative form composed of groups of patterns, as described previously.
Neptune doesn't currently support all Gremlin operators (steps) in its native query engine. So it tries to collapse as many steps as possible down into a single `NeptuneGraphQueryStep`, which contains the declarative logical query plan for all the steps that have been converted. Ideally, all steps are converted. But when a step is encountered that can't be converted, Neptune breaks out of native execution and defers all query execution from that point forward to the TinkerPop steps. It doesn't try to weave in and out of native execution.
After the steps are translated into a logical query plan, Neptune runs a series of query optimizers that rewrite the query plan based on static analysis and estimated cardinalities. These optimizers do things like reorder operators based on range counts, prune unnecessary or redundant operators, rearrange filters, push operators into different groups, and so on.
After an optimized query plan is produced, Neptune creates a pipeline of physical operators that do the work of executing the query. This includes reading data from the statement indices, performing joins of various types, filtering, ordering, and so on. The pipeline produces a solution stream that is then converted back into a stream of TinkerPop Traverser objects.
## Serialization of query results
, ?2, ) . project DISTINCT[?1] {rangeCountEstimate=unknown}],
DFEPatternNode[(?1, ?3, ?4, ?5) . project ALL[?1, ?4] graphFilters=(!= . ), {rangeCountEstimate=unknown}]
}, {rangeCountEstimate=unknown}
]
} [Vertex(?1):GraphStep@[a], Vertex(?4):VertexStep]
} ,
NeptuneTraverserConverterDFEStep
]
+ not converted into Neptune steps: HasStep([name.eq(josh)]),
Neptune steps:
[
NeptuneInterleavingStep {
StepInfo[joinVars=[?7, ?1], frontierElement=Vertex(?7):HasStep, pathElements={a=(last,Vertex(?1):GraphStep@[a])}, listPathElement={}, indexTime=0ms],
DFEStep(Vertex) {
DFENode {
DFEJoinGroupNode[ children={
DFEPatternNode[(?7, ?8, ?9, ?10) . project ALL[?7, ?9] graphFilters=(!= . ), {rangeCountEstimate=unknown}],
DFEPatternNode[(?12, ?11, ?9, ?13) . project ALL[?9, ?12] graphFilters=(!= . ), {rangeCountEstimate=unknown}]
}, {rangeCountEstimate=unknown}
]
} [Vertex(?9):VertexStep, Vertex(?12):VertexStep]
}
}
]
+ not converted into Neptune steps: WherePredicateStep(eq(a)),
Neptune steps:
[
DFECleanupStep
]
Optimized Traversal
===================
Neptune steps:
[
DFEStep(Vertex) {
DFENode {
DFEJoinGroupNode[ children={
DFEPatternNode[(?1, ?3, ?4, ?5) . project ALL[?1, ?4] graphFilters=(!= defaultGraph[526] . ), {rangeCountEstimate=9223372036854775807}]
}, {rangeCountEstimate=unknown}
]
} [Vertex(?1):GraphStep@[a], Vertex(?4):VertexStep]
} ,
NeptuneTraverserConverterDFEStep
]
+ not converted into Neptune steps: NeptuneHasStep([name.eq(josh)]),
Neptune steps:
[
NeptuneMemoryTrackerStep,
NeptuneInterleavingStep {
StepInfo[joinVars=[?7, ?1], frontierElement=Vertex(?7):HasStep, pathElements={a=(last,Vertex(?1):GraphStep@[a])}, listPathElement={}, indexTime=0ms],
DFEStep(Vertex) {
DFENode {
DFEJoinGroupNode[ children={
DFEPatternNode[(?7, ?8, ?9, ?10) . project ALL[?7, ?9] graphFilters=(!= defaultGraph[526] . ), {rangeCountEstimate=9223372036854775807}],
DFEPatternNode[(?12, ?11, ?9, ?13) . project ALL[?9, ?12] graphFilters=(!= defaultGraph[526] . ), {rangeCountEstimate=9223372036854775807}]
}, {rangeCountEstimate=unknown}
]
} [Vertex(?9):VertexStep, Vertex(?12):VertexStep]
}
}
]
+ not converted into Neptune steps: WherePredicateStep(eq(a)),
Neptune steps:
[
DFECleanupStep
]
WARNING: >> [NeptuneHasStep([name.eq(josh)]), WherePredicateStep(eq(a))] << (or one of the children for each step) is not supported natively yet
Predicates
==========
# of predicates: 8
```
See [Information in `explain`](#gremlin-explain-api-results) for a description of the DFE-specific sections in the report.
# Gremlin `profile` API in Neptune
`.
If non-null, the gathered results are returned in a serialized response message in the format specified by this parameter. The number of index operations necessary to produce that response message is reported along with the size in bytes to be sent to the client.
Allowed values are `` or any of the valid MIME type or TinkerPop driver "Serializers" enum values.
```
"application/json" or "GRAPHSON"
"application/vnd.gremlin-v1.0+json" or "GRAPHSON_V1"
"application/vnd.gremlin-v1.0+json;types=false" or "GRAPHSON_V1_UNTYPED"
"application/vnd.gremlin-v2.0+json" or "GRAPHSON_V2"
"application/vnd.gremlin-v2.0+json;types=false" or "GRAPHSON_V2_UNTYPED"
"application/vnd.gremlin-v3.0+json" or "GRAPHSON_V3"
"application/vnd.gremlin-v3.0+json;types=false" or "GRAPHSON_V3_UNTYPED"
"application/vnd.graphbinary-v1.0" or "GRAPHBINARY_V1"
```
+ **profile.indexOps** – `boolean`, allowed values: `TRUE` and `FALSE`, default value: `FALSE`.
If true, shows a detailed report of all index operations that took place during query execution and serialization. Warning: This report can be verbose.
## Sample output of Neptune Gremlin `profile`
, "AUS", ?) . project ?1 .], {estimatedCardinality=1, indexTime=84, hashJoin=true, joinTime=3, actualTotalOutput=1}
PatternNode[(?1, <~label>, ?2=, <~>) . project ask .], {estimatedCardinality=3374, indexTime=29, hashJoin=true, joinTime=0, actualTotalOutput=61}
RepeatNode {
Repeat {
PatternNode[(?3, ?5, ?1, ?6) . project ?1,?3 . IsEdgeIdFilter(?6) . SimplePathFilter(?1, ?3)) .], {hashJoin=true, estimatedCardinality=50148, indexTime=0, joinTime=3}
}
Emit {
Filter(true)
}
LoopsCondition {
LoopsFilter([?1, ?3],eq(2))
}
}, annotations={repeatMode=BFS, emitFirst=true, untilFirst=false, leftVar=?1, rightVar=?3}
}, finishers=[limit(100)], annotations={path=[Vertex(?1):GraphStep, Repeat[Vertex(?3):VertexStep]], joinStats=true, optimizationTime=495, maxVarId=7, executionTime=323}
},
NeptuneTraverserConverterStep
]
Physical Pipeline
=================
NeptuneGraphQueryStep
|-- StartOp
|-- JoinGroupOp
|-- SpoolerOp(100)
|-- DynamicJoinOp(PatternNode[(?1, , "AUS", ?) . project ?1 .], {estimatedCardinality=1, indexTime=84, hashJoin=true})
|-- SpoolerOp(100)
|-- DynamicJoinOp(PatternNode[(?1, <~label>, ?2=, <~>) . project ask .], {estimatedCardinality=3374, indexTime=29, hashJoin=true})
|-- RepeatOp
|-- (Iteration 0) [visited=1, output=1 (until=0, emit=1), next=1]
|-- BindingSetQueue (Iteration 1) [visited=61, output=61 (until=0, emit=61), next=61]
|-- SpoolerOp(100)
|-- DynamicJoinOp(PatternNode[(?3, ?5, ?1, ?6) . project ?1,?3 . IsEdgeIdFilter(?6) . SimplePathFilter(?1, ?3)) .], {hashJoin=true, estimatedCardinality=50148, indexTime=0})
|-- BindingSetQueue (Iteration 2) [visited=38, output=38 (until=38, emit=0), next=0]
|-- SpoolerOp(100)
|-- DynamicJoinOp(PatternNode[(?3, ?5, ?1, ?6) . project ?1,?3 . IsEdgeIdFilter(?6) . SimplePathFilter(?1, ?3)) .], {hashJoin=true, estimatedCardinality=50148, indexTime=0})
|-- LimitOp(100)
Runtime (ms)
============
Query Execution: 392.686
Serialization: 2636.380
Traversal Metrics
=================
Step Count Traversers Time (ms) % Dur
-------------------------------------------------------------------------------------------------------------
NeptuneGraphQueryStep(Vertex) 100 100 314.162 82.78
NeptuneTraverserConverterStep 100 100 65.333 17.22
>TOTAL - - 379.495 -
Repeat Metrics
==============
Iteration Visited Output Until Emit Next
------------------------------------------------------
0 1 1 0 1 1
1 61 61 0 61 61
2 38 38 38 0 0
------------------------------------------------------
100 100 38 62 62
Predicates
==========
# of predicates: 16
WARNING: reverse traversal with no edge label(s) - .in() / .both() may impact query performance
Results
=======
Count: 100
Output: [v[3], v[3600], v[3614], v[4], v[5], v[6], v[7], v[8], v[9], v[10], v[11], v[12], v[47], v[49], v[136], v[13], v[15], v[16], v[17], v[18], v[389], v[20], v[21], v[22], v[23], v[24], v[25], v[26], v[27], v[28], v[416], v[29], v[30], v[430], v[31], v[9...
Response serializer: GRYO_V3D0
Response size (bytes): 23566
Index Operations
================
Query execution:
# of statement index ops: 3
# of unique statement index ops: 3
Duplication ratio: 1.0
# of terms materialized: 0
Serialization:
# of statement index ops: 200
# of unique statement index ops: 140
Duplication ratio: 1.43
# of terms materialized: 393
```
In addition to the query plans returned by a call to Neptune `explain`, the `profile` results include runtime statistics around query execution. Each Join operation is tagged with the time it took to perform its join as well as the actual number of solutions that passed through it.
The `profile` output includes the time taken during the core query execution phase, as well as the serialization phase if the `profile.serializer` option was specified.
The breakdown of the index operations performed during each phase is also included at the bottom of the `profile` output.
Note that consecutive runs of the same query may show different results in terms of run-time and index operations because of caching.
For queries using the `repeat()` step, a breakdown of the frontier on each iteration is available if the `repeat()` step was pushed down as part of a `NeptuneGraphQueryStep`.
## Differences in `profile` reports when DFE is enabled
, "ANC", ?) . project ask .]
PatternNode[(?3, ?5, ?1, ?6) . project ?1,?3 . IsEdgeIdFilter(?6) .]
PatternNode[(?3, <~label>, ?4, <~>) . project ask .]
PatternNode[(?3, ?7, ?8, <~>) . project ?3,?8 . ContainsFilter(?7 in ()) .]
}, annotations={path=[Vertex(?1):GraphStep, Vertex(?3):VertexStep, PropertyValue(?8):PropertiesStep], maxVarId=9}
},
NeptuneTraverserConverterStep
]
Optimized Traversal
===================
Neptune steps:
[
NeptuneGraphQueryStep(PropertyValue) {
JoinGroupNode {
PatternNode[(?1, , "ANC", ?) . project ?1 .], {estimatedCardinality=1}
PatternNode[(?3, ?5, ?1, ?6) . project ?1,?3 . IsEdgeIdFilter(?6) .], {estimatedCardinality=INFINITY}
PatternNode[(?3, ?7=, ?8, <~>) . project ?3,?8 .], {estimatedCardinality=7564}
}, annotations={path=[Vertex(?1):GraphStep, Vertex(?3):VertexStep, PropertyValue(?8):PropertiesStep], maxVarId=9}
},
NeptuneTraverserConverterStep
]
Predicates
==========
# of predicates: 26
WARNING: reverse traversal with no edge label(s) - .in() / .both() may impact query performance
```
The `WARNING` message at the bottom of the output occurs because the `in()` step in the traversal cannot be handled using one of the 3 indexes that Neptune maintains (see [How Statements Are Indexed in Neptune](feature-overview-storage-indexing.md) and [Gremlin statements in Neptune](gremlin-explain-background-statements.md)). Because the `in()` step contains no edge filter, it cannot be resolved using the `SPOG`, `POGS` or `GPSO` index. Instead, Neptune must perform a union scan to find the requested vertices, which is much less efficient.
There are two ways to tune the traversal in this situation. The first is to add one or more filtering criteria to the `in()` step so that an indexed lookup can be used to resolve the query. For the example above, this might be:
```
g.V().has('code','ANC').in('route').values('code')
```
Output from the Neptune `explain` API for the revised traversal no longer contains the `WARNING` message:
```
*******************************************************
Neptune Gremlin Explain
*******************************************************
Query String
============
g.V().has('code','ANC').in('route').values('code')
Original Traversal
==================
[GraphStep(vertex,[]), HasStep([code.eq(ANC)]), VertexStep(IN,[route],vertex), PropertiesStep([code],value)]
Converted Traversal
===================
Neptune steps:
[
NeptuneGraphQueryStep(PropertyValue) {
JoinGroupNode {
PatternNode[(?1, <~label>, ?2, <~>) . project distinct ?1 .]
PatternNode[(?1, , "ANC", ?) . project ask .]
PatternNode[(?3, ?5, ?1, ?6) . project ?1,?3 . IsEdgeIdFilter(?6) . ContainsFilter(?5 in ()) .]
PatternNode[(?3, <~label>, ?4, <~>) . project ask .]
PatternNode[(?3, ?7, ?8, <~>) . project ?3,?8 . ContainsFilter(?7 in ()) .]
}, annotations={path=[Vertex(?1):GraphStep, Vertex(?3):VertexStep, PropertyValue(?8):PropertiesStep], maxVarId=9}
},
NeptuneTraverserConverterStep
]
Optimized Traversal
===================
Neptune steps:
[
NeptuneGraphQueryStep(PropertyValue) {
JoinGroupNode {
PatternNode[(?1, , "ANC", ?) . project ?1 .], {estimatedCardinality=1}
PatternNode[(?3, ?5=, ?1, ?6) . project ?1,?3 . IsEdgeIdFilter(?6) .], {estimatedCardinality=32042}
PatternNode[(?3, ?7=, ?8, <~>) . project ?3,?8 .], {estimatedCardinality=7564}
}, annotations={path=[Vertex(?1):GraphStep, Vertex(?3):VertexStep, PropertyValue(?8):PropertiesStep], maxVarId=9}
},
NeptuneTraverserConverterStep
]
Predicates
==========
# of predicates: 26
```
Another option if you are running many traversals of this kind is to run them in a Neptune DB cluster that has the optional `OSGP` index enabled (see [Enabling an OSGP Index](feature-overview-storage-indexing.md#feature-overview-storage-indexing-osgp)). Enabling an `OSGP` index has drawbacks:
+ It must be enabled in a DB cluster before any data is loaded.
+ Insertion rates for vertices and edges may slow by up to 23%.
+ Storage usage will increase by around 20%.
+ Read queries that scatter requests across all indexes may have increased latencies.
Having an `OSGP` index makes a lot of sense for a restricted set of query patterns, but unless you are running those frequently, it is usually preferable to try to ensure that the traversals you write can be resolved using the three primary indexes.
### Using a large number of predicates