Alexander Watson commited on
Commit
9917d34
1 Parent(s): 214864a

initial commit

Browse files
Files changed (3) hide show
  1. README.md +1 -2
  2. app.py +493 -0
  3. requirements.txt +1 -0
README.md CHANGED
@@ -7,7 +7,6 @@ sdk: streamlit
7
  sdk_version: 1.35.0
8
  app_file: app.py
9
  pinned: false
10
- license: other
11
  ---
12
 
13
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
7
  sdk_version: 1.35.0
8
  app_file: app.py
9
  pinned: false
10
+ license: apache-2.0
11
  ---
12
 
 
app.py ADDED
@@ -0,0 +1,493 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import json
2
+ import logging
3
+ import time
4
+ from io import StringIO
5
+
6
+ import pandas as pd
7
+ import requests
8
+ import streamlit as st
9
+ from datasets import load_dataset
10
+ from gretel_client import Gretel
11
+
12
+ from navigator_helpers import DataAugmentationConfig, DataAugmenter, StreamlitLogHandler
13
+
14
+ # Create a StringIO buffer to capture the logging output
15
+ log_buffer = StringIO()
16
+
17
+ # Create a handler to redirect logging output to the buffer
18
+ handler = logging.StreamHandler(log_buffer)
19
+ handler.setLevel(logging.INFO)
20
+
21
+ # Set up the logger
22
+ logger = logging.getLogger()
23
+ logger.setLevel(logging.INFO)
24
+ logger.addHandler(handler)
25
+
26
+
27
+ SAMPLE_DATASET_URL = "https://gretel-public-website.s3.us-west-2.amazonaws.com/datasets/llm-training-data/dolly-examples-qa-with-context.csv"
28
+ WELCOME_MARKDOWN = """
29
+ Gretel Navigator is an interface designed to help you create high-quality, diverse training data examples through synthetic data generation techniques. It aims to assist in scenarios where you have limited training data or want to enhance the quality and diversity of your existing dataset.
30
+
31
+ ## 🎯 Key Use Cases
32
+
33
+ 1. **Augment Existing Training Data**: Expand your existing training data with additional synthetic examples generated by Gretel Navigator. This can help improve the robustness and generalization of your AI models.
34
+
35
+ 2. **Create Diverse Training or Evaluation Data**: Generate diverse training or evaluation data from plain text or seed examples. This ensures your AI models are exposed to a wide range of scenarios and edge cases during training.
36
+
37
+ 3. **Address Data Limitations**: Generate additional examples to fill gaps in your dataset, particularly for underrepresented classes, rare events, or challenging scenarios. This helps improve your model's ability to handle diverse real-world situations.
38
+
39
+ 4. **Mitigate Bias and Toxicity**: Generate training examples that are unbiased and non-toxic by incorporating diverse perspectives and adhering to ethical guidelines. This promotes fairness and responsible AI development.
40
+
41
+ 5. **Enhance Model Performance**: Improve the performance of your AI models across various tasks by training them on diverse synthetic data generated by Gretel Navigator.
42
+
43
+ ## 🔧 Getting Started
44
+
45
+ To start using Gretel Navigator, you'll need:
46
+
47
+ 1. A Gretel account (free accounts are available).
48
+ 2. Seed text or input/output pairs to create or augment AI training data.
49
+
50
+ ## 📂 Input Data Formats
51
+
52
+ Gretel Navigator supports the following formats for input data:
53
+
54
+ - Existing AI training or evaluation data formats:
55
+ - Input/Output pair format (or instruction/response) with any number of ground truth or "context fields".
56
+ - Plain text data.
57
+ - File formats:
58
+ - Hugging Face dataset
59
+ - CSV
60
+ - JSON
61
+ - JSONL
62
+
63
+ ## 📤 Output
64
+
65
+ Gretel Navigator generates one additional training example per row in the input/output pair format. You can specify requirements for the input and output pairs in the configuration. Run the process multiple times to scale your data to any desired level.
66
+
67
+ ## 🌟 AI Alignment Techniques
68
+
69
+ Gretel Navigator incorporates AI alignment techniques to generate high-quality synthetic data:
70
+
71
+ - Diverse Instruction and Response Generation
72
+ - AI-Aligning-AI Methodology (AAA) for iterative data quality enhancement
73
+ - Quality Evaluation
74
+ - Bias and Toxicity Detection
75
+
76
+ By leveraging these techniques, Gretel Navigator helps you create training data that leads to more robust, unbiased, and high-performing AI models.
77
+
78
+ ---
79
+
80
+ Ready to enhance your AI training data and unlock the full potential of your models? Let's get started with Gretel Navigator! 🚀
81
+ """
82
+
83
+
84
+ def main():
85
+ st.set_page_config(page_title="Gretel", layout="wide")
86
+ st.title("🎨 Gretel Navigator: Enhance Your AI Training Data")
87
+
88
+ with st.expander("Introduction", expanded=False):
89
+ st.markdown(WELCOME_MARKDOWN)
90
+
91
+ st.subheader("Step 1: API Key Validation")
92
+ with st.expander("API Key Configuration", expanded=True):
93
+ api_key = st.text_input(
94
+ "Enter your Gretel API key (https://console.gretel.ai)",
95
+ value="",
96
+ type="password",
97
+ help="Your Gretel API key for authentication",
98
+ )
99
+ if "gretel" not in st.session_state:
100
+ st.session_state.gretel = None
101
+ if st.button("Validate API Key"):
102
+ if api_key:
103
+ try:
104
+ st.session_state.gretel = Gretel(api_key=api_key, validate=True)
105
+ st.success("API key validated. Connection successful!")
106
+ except Exception as e:
107
+ st.error(f"Error connecting to Gretel: {str(e)}")
108
+ else:
109
+ st.warning("Please enter your Gretel API key to proceed.")
110
+ if st.session_state.gretel is None:
111
+ st.stop()
112
+
113
+ st.subheader("Step 2: Data Source Selection")
114
+ with st.expander("Data Source", expanded=True):
115
+ data_source = st.radio(
116
+ "Select data source",
117
+ options=[
118
+ "Upload a file",
119
+ "Select a dataset from Hugging Face",
120
+ "Use a sample dataset",
121
+ ],
122
+ help="Choose whether to upload a file, select a dataset from Hugging Face, or use a sample dataset",
123
+ )
124
+
125
+ df = None
126
+ if data_source == "Upload a file":
127
+ uploaded_file = st.file_uploader(
128
+ "Upload a CSV, JSON, or JSONL file",
129
+ type=["csv", "json", "jsonl"],
130
+ help="Upload the dataset file in CSV, JSON, or JSONL format",
131
+ )
132
+
133
+ if uploaded_file is not None:
134
+ if uploaded_file.name.endswith(".csv"):
135
+ df = pd.read_csv(uploaded_file)
136
+ elif uploaded_file.name.endswith(".json"):
137
+ df = pd.read_json(uploaded_file)
138
+ elif uploaded_file.name.endswith(".jsonl"):
139
+ df = pd.read_json(uploaded_file, lines=True)
140
+ st.success(f"File uploaded successfully: {uploaded_file.name}")
141
+
142
+ elif data_source == "Select a dataset from Hugging Face":
143
+ huggingface_dataset = st.text_input(
144
+ "Hugging Face Dataset Repository",
145
+ help="Enter the name of the Hugging Face dataset repository (e.g., 'squad')",
146
+ )
147
+
148
+ huggingface_split = st.selectbox(
149
+ "Dataset Split",
150
+ options=["train", "validation", "test"],
151
+ help="Select the dataset split to use",
152
+ )
153
+
154
+ if st.button("Load Hugging Face Dataset"):
155
+ if huggingface_dataset:
156
+ try:
157
+ with st.spinner("Loading dataset from Hugging Face..."):
158
+ dataset = load_dataset(
159
+ huggingface_dataset, split=huggingface_split
160
+ )
161
+ df = dataset.to_pandas()
162
+ st.success(
163
+ f"Dataset loaded from Hugging Face repository: {huggingface_dataset}"
164
+ )
165
+ except Exception as e:
166
+ st.error(f"Error loading dataset from Hugging Face: {str(e)}")
167
+ else:
168
+ st.warning("Please provide a Hugging Face dataset repository name.")
169
+
170
+ elif data_source == "Use a sample dataset":
171
+ st.write("Try a sample dataset to get started quickly.")
172
+ if st.button("Try Sample Dataset"):
173
+ try:
174
+ df = pd.read_csv(SAMPLE_DATASET_URL)
175
+ st.success("Sample dataset loaded successfully.")
176
+ except Exception as e:
177
+ st.error(f"Error downloading sample dataset: {str(e)}")
178
+
179
+ if df is not None:
180
+ st.session_state.df = df
181
+ st.session_state.selected_fields = list(df.columns)
182
+ st.write(
183
+ f"Loaded dataset with {len(df)} rows and {len(df.columns)} columns."
184
+ )
185
+ else:
186
+ df = st.session_state.get("df")
187
+
188
+ st.subheader("Step 3: Data Preview and Configuration")
189
+ if df is not None:
190
+ with st.expander("Data Preview", expanded=True):
191
+ st.dataframe(df.head())
192
+
193
+ with st.expander("Input Fields Selection", expanded=True):
194
+ st.write(
195
+ "Select the context fields to provide the LLM access to for generating input/output pairs. This can include existing instructions and responses. All selected fields will be treated as ground truth data."
196
+ )
197
+
198
+ selected_fields = []
199
+ for column in df.columns:
200
+ if st.checkbox(
201
+ column,
202
+ value=column in st.session_state.get("selected_fields", []),
203
+ key=f"checkbox_{column}",
204
+ ):
205
+ selected_fields.append(column)
206
+
207
+ st.session_state.selected_fields = selected_fields
208
+
209
+ with st.expander("Advanced Options", expanded=False):
210
+
211
+ output_instruction_field = st.text_input(
212
+ "Synthetic instruction field",
213
+ value=st.session_state.get("output_instruction_field", "instruction"),
214
+ help="Specify the name of the output field for generated instructions",
215
+ )
216
+ st.session_state.output_instruction_field = output_instruction_field
217
+
218
+ output_response_field = st.text_input(
219
+ "Synthetic response field",
220
+ value=st.session_state.get("output_response_field", "response"),
221
+ help="Specify the name of the output field for generated responses",
222
+ )
223
+ st.session_state.output_response_field = output_response_field
224
+
225
+ num_records = st.number_input(
226
+ "Max number of records from input data to process",
227
+ min_value=1,
228
+ max_value=len(df),
229
+ value=len(df),
230
+ help="Specify the number of records to process",
231
+ )
232
+ st.session_state.num_records = num_records
233
+
234
+ num_instructions = st.number_input(
235
+ "Number of diverse candidate instructions",
236
+ min_value=1,
237
+ value=st.session_state.get("num_instructions", 5),
238
+ help="Specify the number of instructions to generate",
239
+ )
240
+ st.session_state.num_instructions = num_instructions
241
+
242
+ num_responses = st.number_input(
243
+ "Number of diverse candidateresponses",
244
+ min_value=1,
245
+ value=st.session_state.get("num_responses", 5),
246
+ help="Specify the number of responses to generate",
247
+ )
248
+ st.session_state.num_responses = num_responses
249
+
250
+ temperature = st.slider(
251
+ "Temperature",
252
+ min_value=0.0,
253
+ max_value=1.0,
254
+ value=st.session_state.get("temperature", 0.8),
255
+ step=0.1,
256
+ help="Adjust the temperature for response generation",
257
+ )
258
+ st.session_state.temperature = temperature
259
+
260
+ max_tokens_instruction = st.slider(
261
+ "Max tokens (instruction)",
262
+ min_value=1,
263
+ max_value=1024,
264
+ value=st.session_state.get("max_tokens_instruction", 100),
265
+ help="Specify the maximum number of tokens for instructions",
266
+ )
267
+ st.session_state.max_tokens_instruction = max_tokens_instruction
268
+
269
+ max_tokens_response = st.slider(
270
+ "Max tokens (response)",
271
+ min_value=1,
272
+ max_value=1024,
273
+ value=st.session_state.get("max_tokens_response", 100),
274
+ help="Specify the maximum number of tokens for responses",
275
+ )
276
+ st.session_state.max_tokens_response = max_tokens_response
277
+
278
+ with st.expander("Model Configuration", expanded=True):
279
+ st.markdown("### Primary Navigator Models")
280
+
281
+ tabular_models = st.session_state.gretel.factories.get_navigator_model_list(
282
+ "tabular"
283
+ )
284
+ navigator_tabular = st.selectbox(
285
+ "Navigator Tabular",
286
+ options=tabular_models,
287
+ index=st.session_state.get("navigator_tabular_index", 0),
288
+ help="Select the primary Navigator tabular model",
289
+ )
290
+ st.session_state.navigator_tabular_index = tabular_models.index(
291
+ navigator_tabular
292
+ )
293
+
294
+ nl_models = st.session_state.gretel.factories.get_navigator_model_list(
295
+ "natural_language"
296
+ )
297
+ navigator_llm = st.selectbox(
298
+ "Navigator LLM",
299
+ options=nl_models,
300
+ index=st.session_state.get("navigator_llm_index", 0),
301
+ help="Select the primary Navigator LLM",
302
+ )
303
+ st.session_state.navigator_llm_index = nl_models.index(navigator_llm)
304
+
305
+ st.markdown("---")
306
+ st.markdown("### AI Align AI (AAA)")
307
+ st.write(
308
+ "AI Align AI (AAA) is a technique that iteratively improves the quality and coherence of generated outputs by using multiple LLMs for co-teaching and self-teaching. Enabling AAA will enhance the overall quality of the synthetic data, but it may slow down the generation process."
309
+ )
310
+
311
+ use_aaa = st.checkbox(
312
+ "Use AI Align AI (AAA)",
313
+ value=st.session_state.get("use_aaa", True),
314
+ help="Enable or disable the use of AI Align AI.",
315
+ )
316
+ st.session_state.use_aaa = use_aaa
317
+
318
+ co_teach_llms = [] # Initialize co_teach_llms with an empty list
319
+
320
+ if use_aaa:
321
+ st.markdown("#### Navigator Co-teaching LLMs")
322
+ st.write(
323
+ "Select additional Navigator LLMs for co-teaching in AAA. It is recommended to use different LLMs than the primary Navigator LLM for this step."
324
+ )
325
+
326
+ for model in nl_models:
327
+ if model != navigator_llm:
328
+ if st.checkbox(model, value=True, key=f"checkbox_{model}"):
329
+ co_teach_llms.append(model)
330
+ else:
331
+ if st.checkbox(model, value=False, key=f"checkbox_{model}"):
332
+ co_teach_llms.append(model)
333
+ st.session_state.co_teach_llms = co_teach_llms
334
+
335
+ st.markdown("---")
336
+ st.markdown("### Format Prompts")
337
+
338
+ instruction_format_prompt = st.text_area(
339
+ "Instruction Format Prompt",
340
+ value=st.session_state.get(
341
+ "instruction_format_prompt",
342
+ "A well-formulated question or command in everyday English.",
343
+ ),
344
+ help="Specify the format prompt for instructions",
345
+ )
346
+ st.session_state.instruction_format_prompt = instruction_format_prompt
347
+
348
+ response_format_prompt = st.text_area(
349
+ "Response Format Prompt",
350
+ value=st.session_state.get(
351
+ "response_format_prompt",
352
+ "A well-formulated response to the question in everyday English.",
353
+ ),
354
+ help="Specify the format prompt for responses",
355
+ )
356
+ st.session_state.response_format_prompt = response_format_prompt
357
+
358
+ with st.expander("Download SDK Code", expanded=False):
359
+ config_text = f"""
360
+ # Create the data augmentation configuration
361
+ config = DataAugmentationConfig(
362
+ input_fields={st.session_state.selected_fields},
363
+ output_instruction_field="{output_instruction_field}",
364
+ output_response_field="{output_response_field}",
365
+ num_instructions={num_instructions},
366
+ num_responses={num_responses},
367
+ temperature={temperature},
368
+ max_tokens_instruction={max_tokens_instruction},
369
+ max_tokens_response={max_tokens_response},
370
+ api_key=YOUR_GRETEL_API_KEY,
371
+ navigator_tabular="{navigator_tabular}",
372
+ navigator_llm="{navigator_llm}",
373
+ co_teach_llms={co_teach_llms},
374
+ instruction_format_prompt="{instruction_format_prompt}",
375
+ response_format_prompt="{response_format_prompt}"
376
+ )
377
+
378
+ # Create the data augmenter and perform augmentation
379
+ augmenter = DataAugmenter(
380
+ df,
381
+ config,
382
+ use_aaa={use_aaa},
383
+ output_file="results.csv",
384
+ verbose=True,
385
+ )
386
+ new_df = augmenter.augment()
387
+ """
388
+ st.code(config_text, language="python")
389
+ st.download_button(
390
+ label="Download SDK Code",
391
+ data=config_text,
392
+ file_name="data_augmentation_code.py",
393
+ mime="text/plain",
394
+ )
395
+
396
+ start_stop_container = st.empty()
397
+
398
+ col1, col2 = st.columns(2)
399
+ with col1:
400
+ start_button = st.button("🚀 Start")
401
+ with col2:
402
+ stop_button = st.button("🛑 Stop")
403
+
404
+ if start_button:
405
+ with st.expander("Augmentation Results", expanded=True):
406
+ st.subheader("Augmentation Results")
407
+ progress_bar = st.progress(0)
408
+ tab1, tab2 = st.tabs(["Augmented Data", "Logs"])
409
+ with tab1:
410
+ augmented_data_placeholder = st.empty()
411
+ st.info(
412
+ "Click on the 'Logs' tab to see and debug real-time logging for each record as it is generated by the agents."
413
+ )
414
+ with tab2:
415
+ log_container = st.empty()
416
+ logs = []
417
+ max_log_lines = 50
418
+
419
+ def custom_log_handler(msg):
420
+ nonlocal logs
421
+ logs.append(msg)
422
+ if len(logs) > max_log_lines:
423
+ logs = logs[-max_log_lines:]
424
+ log_text = "\n".join(logs)
425
+ log_container.text(log_text)
426
+
427
+ handler = StreamlitLogHandler(custom_log_handler)
428
+ logger = logging.getLogger("navigator_helpers")
429
+ logger.addHandler(handler)
430
+ config = DataAugmentationConfig(
431
+ input_fields=selected_fields,
432
+ output_instruction_field=output_instruction_field,
433
+ output_response_field=output_response_field,
434
+ num_instructions=num_instructions,
435
+ num_responses=num_responses,
436
+ temperature=temperature,
437
+ max_tokens_instruction=max_tokens_instruction,
438
+ max_tokens_response=max_tokens_response,
439
+ api_key=api_key,
440
+ navigator_tabular=navigator_tabular,
441
+ navigator_llm=navigator_llm,
442
+ co_teach_llms=co_teach_llms,
443
+ instruction_format_prompt=instruction_format_prompt,
444
+ response_format_prompt=response_format_prompt,
445
+ )
446
+ augmented_data = []
447
+ start_time = time.time()
448
+ with st.spinner("Generating synthetic data..."):
449
+ for index in range(num_records):
450
+ row = df.iloc[index]
451
+ augmenter = DataAugmenter(
452
+ pd.DataFrame([row]),
453
+ config,
454
+ use_aaa=use_aaa,
455
+ output_file="results.csv",
456
+ verbose=True,
457
+ )
458
+ new_df = augmenter.augment()
459
+ augmented_data.append(new_df)
460
+ augmented_data_placeholder.subheader("Augmented Data")
461
+ augmented_data_placeholder.dataframe(
462
+ pd.concat(augmented_data, ignore_index=True)
463
+ )
464
+ progress = (index + 1) / num_records
465
+ progress_bar.progress(progress)
466
+
467
+ elapsed_time = time.time() - start_time
468
+ records_processed = index + 1
469
+ records_remaining = num_records - records_processed
470
+ est_time_per_record = (
471
+ elapsed_time / records_processed
472
+ if records_processed > 0
473
+ else 0
474
+ )
475
+ est_time_remaining = est_time_per_record * records_remaining
476
+
477
+ progress_text = f"Progress: {progress:.2%} | Records Processed: {records_processed} | Records Remaining: {records_remaining} | Est. Time per Record: {est_time_per_record:.2f}s | Est. Time Remaining: {est_time_remaining:.2f}s"
478
+ progress_bar.text(progress_text)
479
+
480
+ time.sleep(0.1)
481
+ logger.removeHandler(handler)
482
+ st.success("Data augmentation completed!")
483
+ if stop_button:
484
+ st.warning("Augmentation stopped by the user.")
485
+ st.stop()
486
+ else:
487
+ st.info(
488
+ "Please upload a file or select a dataset from Hugging Face to proceed."
489
+ )
490
+
491
+
492
+ if __name__ == "__main__":
493
+ main()
requirements.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ git+https://github.com/gretelai/navigator-helpers.git