MemoryIndex index = new MemoryIndex(withOffsets); for (Map.Entry<String, Collection<Object>> entry : values.entrySet()) { String field = entry.getKey(); if (entry.getValue() instanceof List) { for (Object text : entry.getValue()) { index.addField(field, text.toString(), analyzer); index.addField(field, entry.getValue().toString(), analyzer); return MultiFields.getFields(index.createSearcher().getIndexReader());
/** * Builds a MemoryIndex from a lucene {@link Document} using an analyzer * @param document the document to index * @param analyzer the analyzer to use * @param storeOffsets <code>true</code> if offsets should be stored * @param storePayloads <code>true</code> if payloads should be stored * @return a MemoryIndex */ public static MemoryIndex fromDocument(Iterable<? extends IndexableField> document, Analyzer analyzer, boolean storeOffsets, boolean storePayloads) { return fromDocument(document, analyzer, storeOffsets, storePayloads, 0); }
public void clear() { memoryIndex.reset(); fieldStats.clear(); } /*
public LuceneTokenCounter(Analyzer generalAnalyzer) throws IOException { memoryIndex = new MemoryIndex(); IndexSearcher searcher = memoryIndex.createSearcher(); leafReader = (LeafReader)searcher.getIndexReader(); this.generalAnalyzer = generalAnalyzer; }
/** * Builds a MemoryIndex from a lucene {@link Document} using an analyzer * @param document the document to index * @param analyzer the analyzer to use * @param storeOffsets <code>true</code> if offsets should be stored * @param storePayloads <code>true</code> if payloads should be stored * @param maxReusedBytes the number of bytes that should remain in the internal memory pools after {@link #reset()} is called * @return a MemoryIndex */ public static MemoryIndex fromDocument(Iterable<? extends IndexableField> document, Analyzer analyzer, boolean storeOffsets, boolean storePayloads, long maxReusedBytes) { MemoryIndex mi = new MemoryIndex(storeOffsets, storePayloads, maxReusedBytes); for (IndexableField field : document) { mi.addField(field, analyzer); } return mi; }
public boolean isDeleted(Document document, Object key, OIdentifiable value) { boolean match = false; List<String> strings = deleted.get(value.getIdentity().toString()); if (strings != null) { MemoryIndex memoryIndex = new MemoryIndex(); for (String string : strings) { Query q = engine.deleteQuery(string, value); memoryIndex.reset(); for (IndexableField field : document.getFields()) { memoryIndex.addField(field.name(), field.stringValue(), new KeywordAnalyzer()); } match = match || (memoryIndex.search(q) > 0.0f); } return match; } return match; }
public void add(String field, String content) throws IOException { memoryIndex.addField(field, content, generalAnalyzer); //memoryIndex.addField(field+ALPHA_IDEOGRAPH_SUFFIX, // content, alphaIdeographAnalyzer); count(field); //count(field+ALPHA_IDEOGRAPH_SUFFIX); }
public void buildTermVector(int docid) throws IOException { /* */ Set<String> fieldList = new HashSet<>(); fieldList.add("content"); Document doc = reader.document(docid, fieldList); MemoryIndex mi = MemoryIndex.fromDocument(doc, new StandardAnalyzer()); IndexReader mr = mi.createSearcher().getIndexReader(); Terms t = mr.leaves().get(0).reader().terms("content"); if ((t != null) && (t.size()>0)) { TermsEnum te = t.iterator(); BytesRef term = null; System.out.println(t.size()); while ((term = te.next()) != null) { System.out.println("BytesRef: " + term.utf8ToString()); System.out.println("docFreq: " + te.docFreq()); System.out.println("totalTermFreq: " + te.totalTermFreq()); } } }
private SingletonDocumentBatch(Collection<InputDocument> documents, Similarity similarity) { super(documents, similarity); assert documents.size() == 1; memoryindex.setSimilarity(similarity); for (InputDocument doc : documents) { for (IndexableField field : doc.getDocument()) { memoryindex.addField(field, doc.getAnalyzers()); } } memoryindex.freeze(); reader = (LeafReader) memoryindex.createSearcher().getIndexReader(); }
public class MemoryIndexTest { private static final String DATE_FIELD = "date"; MemoryIndex index = new MemoryIndex(); NumericTokenStream numericTokenStream = new NumericTokenStream(); @Before public void init() { index.addField(DATE_FIELD, numericTokenStream.setIntValue(20141116)); } @Test public void testRange() { Query query = NumericRangeQuery.newIntRange(DATE_FIELD, 20141115, 20141118, true, true); assertTrue(index.search(query) > 0); } }
private boolean matchField(Object iLeft, Object iRight, OLuceneFullTextIndex index, MemoryIndex memoryIndex) throws IOException, ParseException { for (IndexableField field : index.buildDocument(iLeft).getFields()) { memoryIndex.addField(field, index.indexAnalyzer()); } return memoryIndex.search(index.buildQuery(iRight)) > 0.0f; }
private MemoryIndex getOrCreateMemoryIndex(OCommandContext ctx) { MemoryIndex memoryIndex = (MemoryIndex) ctx.getVariable(MEMORY_INDEX); if (memoryIndex == null) { memoryIndex = new MemoryIndex(); ctx.setVariable(MEMORY_INDEX, memoryIndex); } memoryIndex.reset(); return memoryIndex; }
public DocEngineSearcher(MemoryIndex memoryIndex) { super("percolate", memoryIndex.createSearcher()); this.memoryIndex = memoryIndex; }
@Override public Object evaluateRecord(OIdentifiable iRecord, ODocument iCurrentResult, OSQLFilterCondition iCondition, Object iLeft, Object iRight, OCommandContext iContext) { OLuceneFullTextIndex index = involvedIndex(iRecord, iCurrentResult, iCondition, iLeft, iRight); if (index == null) { throw new OCommandExecutionException("Cannot evaluate lucene condition without index configuration."); } MemoryIndex memoryIndex = (MemoryIndex) iContext.getVariable("_memoryIndex"); if (memoryIndex == null) { memoryIndex = new MemoryIndex(); iContext.setVariable("_memoryIndex", memoryIndex); } memoryIndex.reset(); Document doc = index.buildDocument(iLeft); for (IndexableField field : doc.getFields()) { memoryIndex.addField(field.name(), field.stringValue(), index.analyzer(field.name())); } Query query = null; try { query = index.buildQuery(iRight); } catch (Exception e) { throw new OCommandExecutionException("Error executing lucene query.", e); } return memoryIndex.search(query) > 0.0f; }
/** * Equivalent to <code>addField(fieldName, stream, 1.0f)</code>. * * @param fieldName * a name to be associated with the text * @param stream * the token stream to retrieve tokens from */ public void addField(String fieldName, TokenStream stream) { addField(fieldName, stream, 1.0f); }
MemoryIndex mi = MemoryIndex.fromDocument(doc, new StandardAnalyzer()); IndexReader mr = mi.createSearcher().getIndexReader();
memoryIndex = new MemoryIndex(true); memoryIndices[i] = indexDoc(d, analyzer, memoryIndex).createSearcher().getIndexReader();
/** * Builds a MemoryIndex from a lucene {@link Document} using an analyzer * @param document the document to index * @param analyzer the analyzer to use * @param storeOffsets <code>true</code> if offsets should be stored * @param storePayloads <code>true</code> if payloads should be stored * @param maxReusedBytes the number of bytes that should remain in the internal memory pools after {@link #reset()} is called * @return a MemoryIndex */ public static MemoryIndex fromDocument(Document document, Analyzer analyzer, boolean storeOffsets, boolean storePayloads, long maxReusedBytes) { MemoryIndex mi = new MemoryIndex(storeOffsets, storePayloads, maxReusedBytes); for (IndexableField field : document) { mi.addField(field, analyzer); } return mi; }
@Override public Object execute(Object iThis, OIdentifiable iCurrentRecord, Object iCurrentResult, Object[] params, OCommandContext ctx) { OElement element = iThis instanceof OElement ? (OElement) iThis : ((OResult) iThis).toElement(); String indexName = (String) params[0]; OLuceneFullTextIndex index = searchForIndex(ctx, indexName); if (index == null) return false; String query = (String) params[1]; MemoryIndex memoryIndex = getOrCreateMemoryIndex(ctx); List<Object> key = index.getDefinition().getFields().stream().map(s -> element.getProperty(s)).collect(Collectors.toList()); try { for (IndexableField field : index.buildDocument(key).getFields()) { memoryIndex.addField(field, index.indexAnalyzer()); } ODocument metadata = getMetadata(params); OLuceneKeyAndMetadata keyAndMetadata = new OLuceneKeyAndMetadata( new OLuceneCompositeKey(Arrays.asList(query)).setContext(ctx), metadata); return memoryIndex.search(index.buildQuery(keyAndMetadata)) > 0.0f; } catch (ParseException e) { OLogManager.instance().error(this, "error occurred while building query", e); } return null; }
public static MemoryIndex getOrCreateMemoryIndex(OCommandContext ctx) { MemoryIndex memoryIndex = (MemoryIndex) ctx.getVariable(MEMORY_INDEX); if (memoryIndex == null) { memoryIndex = new MemoryIndex(); ctx.setVariable(MEMORY_INDEX, memoryIndex); } memoryIndex.reset(); return memoryIndex; }