1
\$\begingroup\$

I'm very new to Neo4J, and to Jaccard similarity (weighted or non!), so I'd really appreciate someone casting an eye over my code here before I try to use the results.

Jaccard similarity for 2 sets is:

$$ J(A,B) = \frac{\vert A \cap B \vert}{\vert A \cup B \vert} = \frac{\vert A \cap B \vert}{\vert A \vert + \vert B \vert - \vert A \cap B \vert} $$

I calculate that in Neo4j like this:

MATCH (w:Word{toProcess: True})

MATCH (w)-[:NEXT_WORD]->(right:Word), (o:Word)-[:NEXT_WORD]->(right) WHERE w <> o
WITH w, o, count(distinct right.name) as intersection
MATCH (all:Word) WHERE (w)-[:NEXT_WORD]->(all) OR (o)-[:NEXT_WORD]->(all)
WITH w, o, toFloat(intersection)/toFloat(count(distinct all.name)) as rightJaccard

MATCH (w)<-[:NEXT_WORD]-(left:Word), (o)<-[:NEXT_WORD]-(left) WHERE w <> o
WITH w, o, rightJaccard, count(distinct left.name) as intersection
MATCH (all:Word) WHERE (w)<-[:NEXT_WORD]-(all) OR (o)<-[:NEXT_WORD]-(all)
WITH w, o, rightJaccard, toFloat(intersection)/toFloat(count(distinct all.name)) as leftJaccard

RETURN w, o, (rightJaccard + leftJaccard)/2.0 as paradig

(Weighted Jaccard is described here.)

$$ J(x, y) = \frac{\sum_i{\min(x_i,y_i)}}{\sum_i{\max(x_i,y_i)}} $$

I think this code (adapted from the basic Jaccard above) calculates the weighted Jaccard:

MATCH (w:Word{toProcess: True})
MATCH (w)-[rw:NEXT_WORD]->(right:Word), (o:Word)-[ro:NEXT_WORD]->(right) 
WHERE w <> o
WITH w, o, sum(min(rw.weight, ro.weight)) as intersection
MATCH (all:Word) WHERE (w)-[rw1:NEXT_WORD]->(all) AND NOT (o)-[:NEXT_WORD]->(all)
WITH w, o, intersection, sum(rw1.weight) as wSum
MATCH (all:Word) WHERE (w)-[:NEXT_WORD]->(all) AND NOT (o)-[ro1:NEXT_WORD]->(all)
WITH w, o, toFloat(intersection)/toFloat(wSum + sum(ro1.weight) - intersection) as rightJaccard

MATCH (w)<-[rw:NEXT_WORD]-(left:Word), (o)<-[ro:NEXT_WORD]-(left) 
WHERE w <> o
WITH w, o, sum(min(rw.weight, ro.weight)) as intersection
MATCH (all:Word) WHERE (w)<-[rw1:NEXT_WORD]-(all) AND NOT (o)<-[:NEXT_WORD]-(all)
WITH w, o, intersection, sum(rw1.weight) as wSum
MATCH (all:Word) WHERE (w)<-[:NEXT_WORD]-(all) AND NOT (o)<-[ro1:NEXT_WORD]-(all)
WITH w, o, intersection, wSum, 
toFloat(intersection)/toFloat(wSum + sum(ro1.weight) - intersection) as leftJaccard

WITH w, o, (rightJaccard + leftJaccard)/2.0 as sim
MERGE (w)<-[r:RELATED_TO]->(o) SET r.paradig = sim
set w.toProcess = false

I sum all the min weights of the AB intersection set (min weight will be zero outside this set).

Then divide this by the sum of the weights of the individual A and B sets minus the min weights of the intersection set.

Subtracting the min weights of the intersection set, leaves the max weight for the intersection values, and the (by definition) max weight for the difference in the denominator.

I think I'm on firm ground here because expressing it in this way makes clear the link between the weighted version and the 2nd version of the basic Jaccard calculation (as expressed in terms of just the sets and their intersection). But I'd like to be sure my code is right, and that there isn't a better way of doing it in cypher.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.