用于文本的TensorFlow：使用TensorFlow的文本和NLP_tensorflow nlp

Text: "The weather today is neither good nor bad. It's just average."
Sentiment: Neutral

Text: "This product exceeded my expectations. It's worth the price!"
Sentiment: Positive

Text: "I'm quite disappointed with the quality of this product. It doesn't meet my expectations."
Sentiment: Negative

import tensorflow as tf
import numpy as np
import pandas as pd

# Install TensorFlow
!pip install tensorflow
 
# Verify the installation
print(tf.__version__)
 
'
运行
运行

# Load the dataset (example using IMDb Movie Reviews)
data = pd.read_csv('imdb_reviews.csv')
 
# Preprocess the text data
# Tokenization
tokenizer = tf.keras.preprocessing.text.Tokenizer()
tokenizer.fit_on_texts(data['text'])
sequences = tokenizer.texts_to_sequences(data['text'])
 
# Removing stop words
stop_words = set(stopwords.words('english'))
filtered_sequences = []
for seq in sequences:
    filtered_seq = [word for word in seq if word not in stop_words]
    filtered_sequences.append(filtered_seq)
 
# Converting text to numerical representations
max_length = 100
padded_sequences = tf.keras.preprocessing.sequence.pad_sequences(filtered_sequences, maxlen=max_length)
 
'
运行
运行

# Build the sentiment analysis model using LSTM
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(len(tokenizer.word_index) + 1, 64, input_length=max_length),
    tf.keras.layers.LSTM(64),
    tf.keras.layers.Dense(1, activation='sigmoid')
])
 
# Compile the model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
 
# Train the model
model.fit(padded_sequences, data['label'], epochs=10, validation_split=0.2)
 
'
运行
运行

# Evaluate the model
loss, accuracy = model.evaluate(padded_sequences, data['label'])
print("Accuracy:", accuracy)
 
# Perform predictions on new data
new_data = ["This movie is amazing!", "I didn't enjoy the film."]
new_sequences = tokenizer.texts_to_sequences(new_data)
new_padded_sequences = tf.keras.preprocessing.sequence.pad_sequences(new_sequences, maxlen=max_length)
predictions = model.predict(new_padded_sequences)
for i in range(len(new_data)):
    sentiment = "Positive" if predictions[i] > 0.5 else "Negative"
    print("Text:", new_data[i])
    print("Sentiment:", sentiment)
 
'
运行
运行

Text: "I loved the movie! The plot was intriguing, and the acting was superb."
Sentiment: Positive

Text: "The customer service was terrible. I had a horrible experience dealing with them."
Sentiment: Negative

Text: "The weather today is neither good nor bad. It's just average."
Sentiment: Neutral

Text: "Apple Inc. is a technology company."
Predicted Label: ORGANIZATION
Text: "John Smith is a great actor."
Predicted Label: PERSON

import tensorflow as tf
import numpy as np
import pandas as pd
 

# Install TensorFlow
!pip install tensorflow
 
# Verify the installation
print(tf.__version__)
 
'
运行
运行

# Load the CoNLL-2003 dataset (example)
data = pd.read_csv('conll2003_dataset.csv')
 
# Preprocess the text data
# Tokenization
tokenizer = tf.keras.preprocessing.text.Tokenizer()
tokenizer.fit_on_texts(data['text'])
sequences = tokenizer.texts_to_sequences(data['text'])
 
# Labeling and converting text to numerical representations
# (Example using entity labels: B-PERSON, I-PERSON, B-ORG, etc.)
labels = pd.get_dummies(data['label'])
 
# Padding sequences
max_length = max(len(seq) for seq in sequences)
padded_sequences = tf.keras.preprocessing.sequence.pad_sequences(sequences, maxlen=max_length)
 
'
运行
运行

# Build the Named Entity Recognition model using Bidirectional LSTM-CRF
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(len(tokenizer.word_index) + 1, 64, input_length=max_length),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64, return_sequences=True)),
    tf.keras.layers.Dense(len(labels.columns), activation='softmax')
])
 
# Compile the model
model.compile(loss='categorical_crossentropy', optimizer='adam')
 
# Train the model
model.fit(padded_sequences, labels, epochs=10, validation_split=0.2)
 
'
运行
运行

# Evaluate the model
loss = model.evaluate(padded_sequences, labels)
print("Loss:", loss)
 
# Perform predictions on new data
new_data = ["Apple Inc. is a technology company.", "John Smith is a great actor."]
new_sequences = tokenizer.texts_to_sequences(new_data)
new_padded_sequences = tf.keras.preprocessing.sequence.pad_sequences(new_sequences, maxlen=max_length)
predictions = model.predict(new_padded_sequences)
predicted_labels = [labels.columns[np.argmax(pred)] for pred in predictions]
for i in range(len(new_data)):
    print("Text:", new_data[i])
    print("Predicted Label:", predicted_labels[i])
 
'
运行
运行

Text: "I visited Paris last summer and had a great time exploring the Eiffel Tower."
Predicted Labels:
 
"Paris" - LOCATION
"summer" - DATE
"Eiffel Tower" - LOCATION

"Once upon a time, there was a little girl who lived in a magical forest. She had a special power that could turn anything she touched into gold. One day, she stumbled upon a mysterious key that unlocked a hidden door. Curiosity got the best of her, and she entered the door, revealing a world filled with enchanted creatures and endless possibilities."

import tensorflow as tf
import numpy as np
import pandas as pd
 

# Install TensorFlow
!pip install tensorflow
 
# Verify the installation
print(tf.__version__)
 
'
运行
运行

# Load the text corpus or dataset (for example using a text file)
with open('text_corpus.txt', 'r') as file:
    text_corpus = file.read()
 
# Preprocess the text data
# Tokenization
tokenizer = tf.keras.preprocessing.text.Tokenizer()
tokenizer.fit_on_texts([text_corpus])
sequences = tokenizer.texts_to_sequences([text_corpus])
 
# Converting text to numerical representations
num_words = len(tokenizer.word_index) + 1
input_sequences = np.array(sequences)
 
'
运行
运行

# Build the text generation model using an RNN (example using LSTM)
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(num_words, 64, input_length=input_sequences.shape[1]),
    tf.keras.layers.LSTM(128),
    tf.keras.layers.Dense(num_words, activation='softmax')
])
 
# Compile the model
model.compile(loss='sparse_categorical_crossentropy', optimizer='adam')
 
# Train the model
model.fit(input_sequences, np.expand_dims(input_sequences, -1), epochs=10)
 
'
运行
运行

# Generate text using the trained model
seed_text = "Once upon a time"
generated_text = seed_text
for _ in range(50):
    encoded_text = tokenizer.texts_to_sequences([generated_text])[0]
    input_sequence = np.array([encoded_text])
    predicted_word_index = np.argmax(model.predict(input_sequence), axis=-1)
    predicted_word = tokenizer.index_word[predicted_word_index[0][0]]
    generated_text += " " + predicted_word
 
# Print the generated text
print(generated_text)
 
'
运行
运行

"Once upon a time, there was a little girl who lived in a magical forest. She had a special power that could turn anything she touched into gold. One day, she stumbled upon a mysterious key that unlocked a hidden door. Curiosity got the best of her, and she entered the door, revealing a world filled with enchanted creatures and endless possibilities."

# Generate text with a custom prompt
seed_text = "In a galaxy far"
generated_text = seed_text
for _ in range(30):
    encoded_text = tokenizer.texts_to_sequences([generated_text])[0]
    input_sequence = np.array([encoded_text])
    predicted_word_index = np.argmax(model.predict(input_sequence), axis=-1)
    predicted_word = tokenizer.index_word[predicted_word_index[0][0]]
    generated_text += " " + predicted_word
 
# Print the generated text
print(generated_text)
 
'
运行
运行

"In a galaxy far, far away, a group of brave adventurers embarked on a perilous journey to save their dying planet. They encountered strange creatures, battled formidable foes, and forged unlikely alliances. With every step they took, hope grew stronger, and the fate of their world hung in the balance."

import tensorflow as tf
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
 
# Example input text
input_text = """
Text summarization is the process of distilling the most important information from a source text and presenting it in a concise summary. There are two main approaches to text summarization: extractive and abstractive. Extractive summarization involves selecting and combining key phrases or sentences from the original text. Abstractive summarization, on the other hand, aims to generate new sentences that capture the essence of the original text.
In this example, we'll focus on extractive summarization using TensorFlow. We'll use a simple approach based on word frequency and sentence scoring to select the most relevant sentences for the summary.
"""
 
# Preprocess the input text
tokenizer = Tokenizer()
tokenizer.fit_on_texts([input_text])
input_sequences = tokenizer.texts_to_sequences([input_text])
input_sequences = pad_sequences(input_sequences, maxlen=100)
 
# Define the sentence scoring function
def score_sentences(sentences):
    word_frequencies = {}
    for sentence in sentences:
        for word in a sentence:
            if word not in word_frequencies:
                word_frequencies[word] = 1
            else:
                word_frequencies[word] += 1
 
    sentence_scores = {}
    for sentence in sentences:
        for word in a sentence:
            if sentence not in sentence_scores:
                sentence_scores[sentence] = word_frequencies[word]
            else:
                sentence_scores[sentence] += word_frequencies[word]
 
    return sentence_scores
 
# Extract the most important sentences for the summary
sentences = input_text.split('.')
sentence_scores = score_sentences(sentences)
 
summary_sentences = []
summary_length = 2
 
sorted_sentences = sorted(sentence_scores.items(), key=lambda x: x[1], reverse=True)
for i in range(summary_length):
    summary_sentences.append(sorted_sentences[i][0])
 
summary = ' '.join(summary_sentences)
 
print("Input Text:")
print(input_text)
print("Summary:")
print(summary)
 

Text summarization is the process of distilling the most important information from a source text and presenting it in a concise summary. There are two main approaches to text summarization: extractive and abstractive. Extractive summarization involves selecting and combining key phrases or sentences from the original text. Abstractive summarization, on the other hand, aims to generate new sentences that capture the essence of the original text.
 
In this example, we'll focus on extractive summarization using TensorFlow. We'll use a simple approach based on word frequency and sentence scoring to select the most relevant sentences for the summary.

Text summarization is the process of distilling the most important information from a source text and presenting it in a concise summary. In this example, we'll focus on extractive summarization using TensorFlow.
The summary generated by the code consists of the two most relevant sentences from the input text based on the word frequency and sentence scoring approach.