基于Lucene3.5.0怎样从TokenStream获得Token

基于Lucene3.5.0怎样从TokenStream获得Token

通过学习Lucene3.5.0的doc文档,对不同release版本号 lucene版本号的API修改做分析。最后找到了有价值的修改信息。
LUCENE-2302: Deprecated TermAttribute and replaced by a new CharTermAttribute. The change is backwards compatible, so mixed new/old TokenStreams all work on the same char[] buffer independent of which interface they use. CharTermAttribute has shorter method names and implements CharSequence and Appendable. This allows usage like Java’s StringBuilder in addition to direct char[] access. Also terms can directly be used in places where CharSequence is allowed (e.g. regular expressions). 
(Uwe Schindler, Robert Muir)
以上信息可以知道,原来的通过的方法已经不可以提取响应的Token了

StringReader reader = new StringReader(s);
TokenStream ts =analyzer.tokenStream(s, reader);
TermAttribute ta = ts.getAttribute(TermAttribute.class);

通过分析Api文档信息 可知,CharTermAttribute已经成为替换TermAttribute的接口因此我编写了一个样例来更好的从TokenStream中提取Token

package com.segment;

import java.io.StringReader;
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.Token;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
import org.apache.lucene.analysis.tokenattributes.TermAttribute;
import org.apache.lucene.util.AttributeImpl;
import org.wltea.analyzer.lucene.IKAnalyzer;


public class Segment {
	public static String show(Analyzer a, String s) throws Exception {

		StringReader reader = new StringReader(s);
		TokenStream ts = a.tokenStream(s, reader);
		String s1 = "", s2 = "";
		boolean hasnext= ts.incrementToken();
		//Token t = ts.next();
		while (hasnext) {
			//AttributeImpl ta = new AttributeImpl();
			CharTermAttribute ta = ts.getAttribute(CharTermAttribute.class);
			//TermAttribute ta = ts.getAttribute(TermAttribute.class);
			
			s2 = ta.toString() + " ";
			s1 += s2;
			hasnext = ts.incrementToken();
		}
		return s1;
	}

	public String segment(String s) throws Exception {
		Analyzer a = new IKAnalyzer();
		return show(a, s);
	}
	public static void main(String args[])
	{
		String name = "我是俊杰,我爱编程,我的測试用例";
		Segment s = new Segment();
		String test = "";
		try {
			System.out.println(test+s.segment(name));
		} catch (Exception e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}

}

版权声明:本文内容由互联网用户自发贡献,该文观点仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 举报,一经查实,本站将立刻删除。

发布者:全栈程序员-用户IM,转载请注明出处:https://javaforall.cn/109513.html原文链接:https://javaforall.cn

【正版授权,激活自己账号】: Jetbrains全家桶Ide使用,1年售后保障,每天仅需1毛

【官方授权 正版激活】: 官方授权 正版激活 支持Jetbrains家族下所有IDE 使用个人JB账号...

(0)


相关推荐

发表回复

您的电子邮箱地址不会被公开。

关注全栈程序员社区公众号