Train NER model using custom IOB corpus using NLTK, train_chunerk.py
0
I made the custom IBO corpus for custom NER model.
The corpus format is below.
Liver NNP B-Loc
, , O
needle JJ O
biopsy NN O
: : O
Chronic JJ B-Pathol
hepatitis NN I-Pathol
, , O
[command]
python train_chunker.py ../iob_corpus --reader nltk.corpus.reader.chunked.ChunkedCorpusReader
[result]
D:Anaconda3libsite-packagessklearnfeature_extractiontext.py:17: Deprecatio
nWarning: Using or importing the ABCs from 'collections' instead of from 'collec
tions.abc' is deprecated, and in 3.8 it will stop working
from collections import Mapping, defaultdict
D:Anaconda3libsite-packagessklearnensembleweight_boosting.py:29: Deprecati
onWarning: numpy.core.umath_tests is an internal NumPy module and should not be
imported. It will be removed in a future NumPy release.
from numpy.core.umath_tests import inner1d
loading ../iob_corpus
Traceback (most recent call last):
File "train_chunker.py", line 145, in <module>
nchunks = len(chunk_trees)
File "D:Anaconda3libsite-packagesnltkcorpusreaderutil.py", line 380, in
__len__
for tok in self.iterate_from(self._offsets[-1]): pass
File "D:Anaconda3libsite-packagesnltkcorpusreaderutil.py", line 402, in
iterate_from
for tok in piece.iterate_from(max(0, start_tok-offset)):
File "D:Anaconda3libsite-packagesnltkcorpusreaderutil.py", line 296, in
iterate_from
tokens = self.read_block(self._stream)
File "D:Anaconda3libsite-packagesnltkcorpusreaderchunked.py", line 179,
in read_block
target_tagset=self._target_tagset)
File "D:Anaconda3libsite-packagesnltkchunkutil.py", line 355, in tagstr2
tree
raise ValueError('Expected ] at char {:d}'.format(len(s)))
ValueError: Expected ] at char 8
I don't know the cause of the error.
Is the problem with the format of the corpus?
Similar errors were made when the IOB checker corpus was made as follows.
Liver B-Loc
, O
needle O
biopsy O
: O
Focal O
nodular O
hyperplasia O
nltk ner tagged-corpus
asked Nov 23 '18 at 6:09
Yoonhee ParkYoonhee Park
12
add a comment |
0
I made the custom IBO corpus for custom NER model.
The corpus format is below.
Liver NNP B-Loc
, , O
needle JJ O
biopsy NN O
: : O
Chronic JJ B-Pathol
hepatitis NN I-Pathol
, , O
[command]
python train_chunker.py ../iob_corpus --reader nltk.corpus.reader.chunked.ChunkedCorpusReader
[result]
D:Anaconda3libsite-packagessklearnfeature_extractiontext.py:17: Deprecatio
nWarning: Using or importing the ABCs from 'collections' instead of from 'collec
tions.abc' is deprecated, and in 3.8 it will stop working
from collections import Mapping, defaultdict
D:Anaconda3libsite-packagessklearnensembleweight_boosting.py:29: Deprecati
onWarning: numpy.core.umath_tests is an internal NumPy module and should not be
imported. It will be removed in a future NumPy release.
from numpy.core.umath_tests import inner1d
loading ../iob_corpus
Traceback (most recent call last):
File "train_chunker.py", line 145, in <module>
nchunks = len(chunk_trees)
File "D:Anaconda3libsite-packagesnltkcorpusreaderutil.py", line 380, in
__len__
for tok in self.iterate_from(self._offsets[-1]): pass
File "D:Anaconda3libsite-packagesnltkcorpusreaderutil.py", line 402, in
iterate_from
for tok in piece.iterate_from(max(0, start_tok-offset)):
File "D:Anaconda3libsite-packagesnltkcorpusreaderutil.py", line 296, in
iterate_from
tokens = self.read_block(self._stream)
File "D:Anaconda3libsite-packagesnltkcorpusreaderchunked.py", line 179,
in read_block
target_tagset=self._target_tagset)
File "D:Anaconda3libsite-packagesnltkchunkutil.py", line 355, in tagstr2
tree
raise ValueError('Expected ] at char {:d}'.format(len(s)))
ValueError: Expected ] at char 8
I don't know the cause of the error.
Is the problem with the format of the corpus?
Similar errors were made when the IOB checker corpus was made as follows.
Liver B-Loc
, O
needle O
biopsy O
: O
Focal O
nodular O
hyperplasia O
nltk ner tagged-corpus
asked Nov 23 '18 at 6:09
Yoonhee ParkYoonhee Park
12
add a comment |
0
0
0
I made the custom IBO corpus for custom NER model.
The corpus format is below.
Liver NNP B-Loc
, , O
needle JJ O
biopsy NN O
: : O
Chronic JJ B-Pathol
hepatitis NN I-Pathol
, , O
[command]
python train_chunker.py ../iob_corpus --reader nltk.corpus.reader.chunked.ChunkedCorpusReader
[result]
D:Anaconda3libsite-packagessklearnfeature_extractiontext.py:17: Deprecatio
nWarning: Using or importing the ABCs from 'collections' instead of from 'collec
tions.abc' is deprecated, and in 3.8 it will stop working
from collections import Mapping, defaultdict
D:Anaconda3libsite-packagessklearnensembleweight_boosting.py:29: Deprecati
onWarning: numpy.core.umath_tests is an internal NumPy module and should not be
imported. It will be removed in a future NumPy release.
from numpy.core.umath_tests import inner1d
loading ../iob_corpus
Traceback (most recent call last):
File "train_chunker.py", line 145, in <module>
nchunks = len(chunk_trees)
File "D:Anaconda3libsite-packagesnltkcorpusreaderutil.py", line 380, in
__len__
for tok in self.iterate_from(self._offsets[-1]): pass
File "D:Anaconda3libsite-packagesnltkcorpusreaderutil.py", line 402, in
iterate_from
for tok in piece.iterate_from(max(0, start_tok-offset)):
File "D:Anaconda3libsite-packagesnltkcorpusreaderutil.py", line 296, in
iterate_from
tokens = self.read_block(self._stream)
File "D:Anaconda3libsite-packagesnltkcorpusreaderchunked.py", line 179,
in read_block
target_tagset=self._target_tagset)
File "D:Anaconda3libsite-packagesnltkchunkutil.py", line 355, in tagstr2
tree
raise ValueError('Expected ] at char {:d}'.format(len(s)))
ValueError: Expected ] at char 8
I don't know the cause of the error.
Is the problem with the format of the corpus?
Similar errors were made when the IOB checker corpus was made as follows.
Liver B-Loc
, O
needle O
biopsy O
: O
Focal O
nodular O
hyperplasia O
nltk ner tagged-corpus
asked Nov 23 '18 at 6:09
Yoonhee ParkYoonhee Park
12
I made the custom IBO corpus for custom NER model.
The corpus format is below.
Liver NNP B-Loc
, , O
needle JJ O
biopsy NN O
: : O
Chronic JJ B-Pathol
hepatitis NN I-Pathol
, , O
[command]
python train_chunker.py ../iob_corpus --reader nltk.corpus.reader.chunked.ChunkedCorpusReader
[result]
D:Anaconda3libsite-packagessklearnfeature_extractiontext.py:17: Deprecatio
nWarning: Using or importing the ABCs from 'collections' instead of from 'collec
tions.abc' is deprecated, and in 3.8 it will stop working
from collections import Mapping, defaultdict
D:Anaconda3libsite-packagessklearnensembleweight_boosting.py:29: Deprecati
onWarning: numpy.core.umath_tests is an internal NumPy module and should not be
imported. It will be removed in a future NumPy release.
from numpy.core.umath_tests import inner1d
loading ../iob_corpus
Traceback (most recent call last):
File "train_chunker.py", line 145, in <module>
nchunks = len(chunk_trees)
File "D:Anaconda3libsite-packagesnltkcorpusreaderutil.py", line 380, in
__len__
for tok in self.iterate_from(self._offsets[-1]): pass
File "D:Anaconda3libsite-packagesnltkcorpusreaderutil.py", line 402, in
iterate_from
for tok in piece.iterate_from(max(0, start_tok-offset)):
File "D:Anaconda3libsite-packagesnltkcorpusreaderutil.py", line 296, in
iterate_from
tokens = self.read_block(self._stream)
File "D:Anaconda3libsite-packagesnltkcorpusreaderchunked.py", line 179,
in read_block
target_tagset=self._target_tagset)
File "D:Anaconda3libsite-packagesnltkchunkutil.py", line 355, in tagstr2
tree
raise ValueError('Expected ] at char {:d}'.format(len(s)))
ValueError: Expected ] at char 8
I don't know the cause of the error.
Is the problem with the format of the corpus?
Similar errors were made when the IOB checker corpus was made as follows.
Liver B-Loc
, O
needle O
biopsy O
: O
Focal O
nodular O
hyperplasia O
nltk ner tagged-corpus
nltk ner tagged-corpus
asked Nov 23 '18 at 6:09
Yoonhee ParkYoonhee Park
12
asked Nov 23 '18 at 6:09
Yoonhee ParkYoonhee Park
12
edited Nov 23 '18 at 6:21
Yoonhee Park
asked Nov 23 '18 at 6:09
Yoonhee ParkYoonhee Park
12
asked Nov 23 '18 at 6:09
Yoonhee ParkYoonhee Park
12
asked Nov 23 '18 at 6:09
Yoonhee ParkYoonhee Park
12
12
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53441420%2ftrain-ner-model-using-custom-iob-corpus-using-nltk-train-chunerk-py%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53441420%2ftrain-ner-model-using-custom-iob-corpus-using-nltk-train-chunerk-py%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
