<jupyter_start><jupyter_text>Digital Commons: Designing Graphs and Charts for PublicationPurpose: In preparation for the "Digital Commons: Designing Graphs and Charts for Publication" workshop on 20220225T12:00, Feb 25 @ Noon- The initial request for Science Engineering Libraries (SEL) involvement in this Digital Commons workshop can be found in "Initial_Digital_Commons_Workshop_Figures_and_graphs_Request" In response to [<NAME>](https://www.vanderbilt.edu/chemistry/faculty/macdonald.php), a chemistry professor and member of the Digital Commons assistant director search committee. Dr. MacDonanld requested for some resources on designing figures and graphs for scientific papers. The premise is providing faculty some basic design skills for creating figures and graphs for publication.Following <NAME> request, they informed SEL that their group is joining the workshop and are aspiring to publish in [ACS Nano](https://pubs.acs.org/page/ancac3/about.html) or [Nano Letters](https://pubs.acs.org/page/nalefd/about.html)"Josh and Francisco! I am so glad you are doing this and I am going to try and get several people from my group to join. I thought it might be helpful to mention where we like to publish and aspire to publish: ACS Nano, or Nano Letters would be great examples. Your presentation of your data has to be Perfecto! (kisses fingers) for publication there. Janet"With that in mind SEL used [Mendeley Data/Digital Commons Data](https://data.mendeley.com/about) to find datasets from ACS Nano or Nano Letters. It's possible to filer by journal in Mendeley Data (e.g., [ACS Nano](https://data.mendeley.com/journal/1936086X) ), but i needed to google "Mendeley Data ACS Nano" to reach these filters. Unfortunately, there is none for Nano Letters. I did try editing the url "suffix" because if you notice the url for the Mendeley Data ACS Nano filter (i.e., https://data.mendeley.com/journal/1936086X), the suffix is ACS Nano's issn. Doing so returned ["Journal not found"](https://data.mendeley.com/journal/15306984) issue. Explanation: Francisco inputted incorrect issn or there is no Nano Letters indexed in the Mendeley Data Database. *Why did Francisco look for a dataset from an ACS Nano paper?*The goal is to use datasets from recently published articles in these journals (i.e., ACS Nano and Nano Letters) and recreate their infographics with Tableau. In doing so we can best present the strenghts of Tableau and meet MacDonalds request of software that can create publication quality graphs. So, if we can successfully replicate all, or at least a majority of, the graphs from an ACS Nano or Nano Letters paper with a tool like Tableau the attendees can leave feeling confident it's a tool worth looking into. Currently, Francisco believes there might be some bias against using Tableau for "hard science" as it can come off seeming low intelligent but in reality we at SEL value it's practicality along with being a more suitable tool for a workshop which is time sensitive. *Tools used in the workshop?*The two primary tools SEL considered for "Digital Commons: Designing Graphs and Charts for Publication"  - [Tableau](https://www.tableau.com/products/desktop)     - Tableau since it's a software that is great for the limitation of a workshop (i.e., time, learning curve, easy-to-follow) and     - the [Office of Data and Strategic Analytics](https://www.vanderbilt.edu/pie/our-team/) is considering getting an institutional license for Tableau which will have added benefits to use the tool like local figure saving and server options.- [Overleaf](https://www.overleaf.com/about).     - OverLeaf for chart/table creation. There is also the benefit that Vanderbilt has an institutional license for Overleaf and     Additionally,SEL will reference and recommened Steven Baskauf's [Python courses](https://heardlibrary.github.io/digital-scholarship/script/python/wg/) from [Digital Scholarshipand Communications](https://www.library.vanderbilt.edu/disc/)SEL will also propose a more indepth collaboration with labs who are interested in integrating more robust software and data management practices through sharing what can be accomplished through openSource software like python and it's myriad of packages and environments (e.g., Pandas, Altair, JupyterNotebook) that exist. We can share a jupyter notebook providing interactive graphs that run on JavaScript in the backend (i.e., html/web compatability) incase there are individuals who are keen on improving their labs site to reflect the work they accomplish. We wil need to provide pros and cons to using tools like Jupyter and these python packages and maybe provide an explanation for why we didnt demo them for this workshop (e.g., time constraint, high learning curve), but to not mention them would be a disservice to our userbase. Francisco has also reached out to an old professor for the figure portion. Francisco is lacking on making figures themselves and working with image data but considering they're omnipresent within Chem related publications it'll be a benefit for SEL if we can become familiar with the available resources that exist.  Why are we in a Jupyter Notebook?To use Tableau the data must be in the correct format. A "correct" format that has worked before is "tidying" tabular data and then running that in Tableau.What does correct format in Tableau this mean? Does it mean data need to be in "Third Normal Form"Third normal form database normalization standards (i.e.,  normalizing principles to reduce the duplication of data, avoid data anomalies, ensure referential integrity, and simplify data management).FRANCISCO READ WHAT IS THE FORMAT TABLEAU LIKES DATA IN FOR THEIR VISUALS AND WRITE THAT INFORMATION HERE OR IN ANOTHER DOCUMENT. Idea: use some of the data as it is in the dataset into tableau for a compare and contrast demonstration?? Cons: makes it look like you might not know what you are doing if the figures arent coming through. depends on how you phrase this. Pro: opportunity to discuss what i means to understand the structure and semantic involved in datasets. - https://help.tableau.com/current/pro/desktop/en-us/data_structure_for_analysis.htm<NAME> is using this jupyter notebook to ["tidy"](http://www.jeannicholashould.com/tidy-data-in-python.html:~:text=A%20tidy%20version%20of%20this%20dataset%20is%20one,from%20a%20wide%20format%20to%20a%20long%20format.) data from ["Sequence-Dependent Self-Assembly andStructural Diversity of Islet Amyloid Polypeptide-Derived β‑Sheet Fibrils"](https://doi.org/10.1021/acsnano.7b02325) so we can then use Tableau without with data structured to compliment the software. Another benefit of Tableau that might be worth mentioning is that it can almost "teach" a user (e.g., grad student) the datastructure that visualization software prefers.  can we use pathlib to print out all the unique file extentions that exists nested in a folder? Look into this. it can provide us a glimpse of the data for the figures in a publication. For example, is we find a csv or excel file we are working with tabular data that can be mututated easily. That can provide us with a quick grasp of which figures we should focus on to reproduce in the workshop.  Figures with graphs in [Sequence-Dependent Self-Assembly and Structural Diversity of Islet Amyloid Polypeptide-Derived β-Sheet Fibrils](https://doi.org/10.1021/acsnano.7b02325):- Figure 2    - d and e- Figure 4    - a, b, and c- Figure 5     - e, f, and g- Figure 6    - a    - this is the most promising figure so we'll work on this one- Figure 7    - c and d- Figure 8    - d, e, and f  Why use Altair Visualization package? altair is a python-wrapper for Vega-lite. https://towardsdatascience.com/consistently-beautiful-visualizations-with-altair-themes-c7f9f889602   [Altair Tutorial](https://altair-viz.github.io/altair-tutorial/README.html)  Why use the Python Pathlib library?File paths and pathlibYou may or may not have seen that macs and windows have different formats for file paths. This means that if you are trying to programmatically generate a path that will work on multiple operating systems, you have to do some strange things.Also, even if you are not worring about that, trying to just get the file name out of an arbitrarily deep file path usually would result in a ton of strange string manipulations. Also, if you were trying to get the root name of a file out so you could add a different file extension, you would have to do some even worse manipulation.This is where pathlib comes in. It understands the file system from an object oriented perspective, so you are operating on and extracting information from the file path using object attributes and methods rather than manipulating a string without conceptuale awareness of the core content meaning.First, import pathlib. This will give you access to the Path() function. You can pass this function a full file name or a directory name. Passing it a directory will give you access to the glob() method for searching for file paths. Pathlib and LabsTheoretically, if you have data in various different directories you can use pathlib to open them while not needing the jupyternotebook to exists in that directory to access the data. When you share your research data you can thus maintain your file organization while the user can run the cells. Useful Links to read more- [Python Standard Library Lecture](https://elliewix.gitbook.io/is597-python-standard-library/weekly-lectures/week-3file-paths-and-pathlib)- [Pathlib Documentation](https://docs.python.org/3/library/pathlib.html)- [RealPython: Python 3's pathlib Module: Taming the File System](https://realpython.com/python-pathlib/moving-and-deleting-files)- [Using Python’s Pathlib Module](https://pbpython.com/pathlib-intro.html)<jupyter_code>parent = pathlib.Path(".") #parent directory of the current working directory this jupyter notebook is in
results = list(parent.glob('*')) # list of everything that exists in the directory/folder that this jupyter notebook is in
results
Figure06APath = pathlib.Path('Figure06A.jpeg')
Figure06APath
paper01datasetPath = pathlib.Path('Compare_side_chain_ThT_integrate_data.csv')
paper01datasetPath
df01 = pd.read_csv(paper01datasetPath, index_col=False)
df01
Figure6A = pd.melt(df01, # making a 'tidy' dataframe
                       ["Wavelength"],
                       var_name="AminoAcidSideChains",
                       value_name="Flourescence")
Figure6A.head(10) 
# dataframe (i.e., table) of the data needed for Figure06A
# .head(10) returns the first 10 rows in the dataframe<jupyter_output><empty_output><jupyter_text>Restructuring the tables into a format suitable for data visualization software and analysis. Original concept poplularized from <NAME> ["Tidy Data"](http://vita.had.co.nz/papers/tidy-data.pdf) paperThe defined format makes it easier to query and filter the data. This approach increases reusability of the data and also makes it easier to share a dataset with other for analysts.<jupyter_code>im = Image(filename=Figure06APath) # read-in/post figure06A from the paper
im # the wavelenghtXFlourescence is the graph we want to reporduce
Figure6A['AminoAcidSideChains'].unique() # what are the unique entries in this column?
Figure6A.shape # row and column
Figure6A.columns # returns a "list" of the columns in the dataframe<jupyter_output><empty_output><jupyter_text>[Data Transformations](https://iliatimofeev.github.io/altair-viz.github.io/user_guide/transform.html)It is often necessary to transform or filter data in the process of visualizing it. In Altair you can do this one of two ways:1. Before the chart definition, using standard Pandas data transformations.2. Within the chart definition, using Vega-Lite’s data transformation tools.for this example we used Pandas to transform our data beforehand<jupyter_code>Figure6A['Flourescence_x10\u00b3'] = Figure6A['Flourescence']/1000 # adding a data transformation column to the dataframe
Figure6A.info()
im<jupyter_output><empty_output><jupyter_text>General chart<jupyter_code>alt.Chart(Figure6A).mark_line().encode(
    alt.X("Wavelength:Q", title="Wavelength / nm"),
    alt.Y("Flourescence:Q", title="Fluorescence (x10\u00b3) /a.u."), 
    alt.Color('AminoAcidSideChains:N', title="Amino Acid Side Chains"),
    tooltip=['AminoAcidSideChains']
)<jupyter_output><empty_output><jupyter_text>Edit legend and adding subscript using unicodeWe can use [unicode](https://www.geeksforgeeks.org/how-to-print-superscript-and-subscript-in-python/) to add subscript in our graph text (e.g., axis, titles, legengs)<jupyter_code>alt.Chart(Figure6A).mark_line().encode(
    alt.X("Wavelength:Q", title="Wavelength / nm"),
    alt.Y("Flourescence_x10\u00b3:Q", title="Fluorescence (x10\u00b3) /a.u."),
    alt.Color('AminoAcidSideChains:N', title="Amino Acid Side Chains"),
    tooltip=['AminoAcidSideChains']
).configure_legend(
    strokeColor='gray',
    fillColor='#EEEEEE',
    padding=10,
    cornerRadius=10,
    orient='top-right'
)<jupyter_output><empty_output><jupyter_text>Copied the code to move and change the legend from altiar documentation [Top-Level Chart Configuration](https://altair-viz.github.io/user_guide/configuration.htmlconfig-title) "allowing the user to customize the apperance of chart legends (e.g., location, fonts, bounding boxes, colors, etc.)  Exact Graph Replicated<jupyter_code>aminoAcids = ['NN-SS 1.5mM-F02', 'NN-TT 1.5mM-F04', 'NN-VV 1.5mM-F05', 'NN-AA 1.5mM-F08']

Figure06A= alt.Chart(Figure6A, title='a)').mark_line().encode(
    alt.X("Wavelength:Q", title="Wavelength / nm"), # , scale=alt.Scale(domain=[400, 660])
    alt.Y("Flourescence_x10\u00b3:Q", title="Fluorescence (x10\u00b3) /a.u.", scale=alt.Scale(domain=[0, 5])),
    alt.Color('AminoAcidSideChains:N', title="Amino Acid Side Chains"),
    tooltip=['AminoAcidSideChains']
).transform_filter(
    alt.FieldOneOfPredicate(field='AminoAcidSideChains', oneOf=aminoAcids)
).configure(
    font='Arial' #Default font for all text marks, titles, and labels.
).configure_title(
    anchor='start',
    fontSize=12
).configure_legend(
    orient='top-right'
).configure_axis(
    grid=False,
    labelFontSize=12,
    titleFontSize=12
).configure_view(
    strokeWidth=0
)


Figure06A # you can save the graph as a PNG by clicking the ellipses in the top-rigth corner
im<jupyter_output><empty_output><jupyter_text>We have sucessfully managed to replicate Figure6a as close as possible, but you might notice that we had to filter out three amino acid chains since the the original image only has 4. However, we do not have the names of the amino acids only what was recorded in the dataset. For example, NN-VV 1.5mM-F05 = Ac-NNFGAILVV We could go back to our dataframe and using pandas rename our Amino Acid Chains to match what is showin the the original chart. However, with no dataDictionary in the original and improper documentation this is not worth the effort for our purposes.    Ways of creating templates that can make plots consistent - [Customizing Visualizations](https://altair-viz.github.io/user_guide/customization.html?highlight=themecustomizing-visualizations)    - This section outlines some of the ways users can make adjustments to visualizations    - for example: setting themes, adjusting axis limits (e.g., scale/domain), adjust axis labels (e.g., format, titles, ), adjust legends, removing chart border, customize colors, Adjusting the width of Bar Marks and chart, adjusting chart size- [Altair Documentaiton: Top-level-Chart-Configuration](https://altair-viz.github.io/user_guide/configuration.htmltop-level-chart-configuration)    - Think of the configure_*() methods as setting a chart theme (i.e., set the default styles for the entire chart)- [Altair Themes](https://altair-viz.github.io/user_guide/configuration.htmlaltair-themes)    - A theme, in altair, is a set of chart configurations applied globally each python session. This means you can **produce similar-looking visualizations consistently**    - You create a theme and then apply chart configurations globally within any Python session    - done using the alt.theme object    - Since Altair uses Vega in their backend programing you can use any of the preconstructed themes from [Vega Themes](https://github.com/vega/vega-themes/included-themes). If you find one that is close to what you want, you can copy the code and adjust as fit. Read more on making a a theme here: [define a custom theme](https://altair-viz.github.io/user_guide/configuration.htmldefining-a-custom-theme)- [Consistently Beautiful Visualizations with Altair Themes](https://towardsdatascience.com/consistently-beautiful-visualizations-with-altair-themes-c7f9f889602)    - A blog focused on using themes and how you can code one into your script- [established pattern for importing Vega themes into Altair](https://github.com/altair-viz/altair/issues/1333)    - a Github tread discussing how to import Themes     - the most consistent way is writing `alt.themes.enable('string of theme name goes here')`- [Example Code of how to construct an Altair Chart Theme](https://pmbaumgartner.github.io/streamlitopedia/theming-altair.html)    - code examples of someone who has used Altair to make themes.    - it's a collection of free to use of tools, code snippets, and ideas that they found helpful when making [streamlit](https://streamlit.io/) apps. Even discuss markup language which is used in jupyter notbeook<jupyter_code># example of using a pre-constructed theme in your visual 
# Using the with statement so you dont apply the theme to all charts but only the one in that cell
# If you would like to use any theme just for a single chart, you can use the `with` statement to enable a temporary theme:

with alt.themes.enable('ggplot2'):
    Figure06A= alt.Chart(Figure6A, title='a)').mark_line().encode(
    alt.X("Wavelength:Q", title="Wavelength / nm"),
    alt.Y("Flourescence_x10\u00b3:Q", title="Fluorescence (x10\u00b3) /a.u."),
    alt.Color('AminoAcidSideChains:N', title="Amino Acid Side Chains"),
    tooltip=['AminoAcidSideChains']
    ).transform_filter(
        alt.FieldOneOfPredicate(field='AminoAcidSideChains', oneOf=aminoAcids)
    )
    # Figure06A.show() # This command will block the Python interpreter until the browser window containing the chart is closed.
    Figure06A.display()
    # Note: that calling .display() on multiple charts is how you can display multiple charts in a single cell / image
    # more infor here: 
# example of using a pre-constructed theme in your visual to apply globally

alt.themes.enable('latimes')
Figure06A= alt.Chart(Figure6A, title='a)').mark_line().encode(
alt.X("Wavelength:Q", title="Wavelength / nm"),
alt.Y("Flourescence_x10\u00b3:Q", title="Fluorescence (x10\u00b3) /a.u."),
alt.Color('AminoAcidSideChains:N', title="Amino Acid Side Chains"),
tooltip=['AminoAcidSideChains']
).transform_filter(
    alt.FieldOneOfPredicate(field='AminoAcidSideChains', oneOf=aminoAcids)
)

Figure06A

    
# If you want to restore the default theme, use:
alt.themes.enable('default')
# In general, themes effect the aesthethics of the graphs. 
# let's use @ipywidgets "decorators" to make quick interactive functions
# to flip between different pre-created themes
@ipywidgets.interact(name = ['Linda', 'Tina', 'Louise']) # list of names options in the dropdown menu
def print_name(name):
    print(name) # just a simple print out. 
    
# we'll be using @ipywidgets to add a dropdown menu to change the chart/graph theme 
type(alt.vegalite.v4.theme) 
# i wonder how we can get a list of the available themes in any altair version?
# this module might be the key to finding a way to get a list of them but i dont know
@ipywidgets.interact(theme = ['vox','default','dark','ggplot2','latimes','dark', 'quartz','fivethirtyeight','urbaninstitute'])
# the list of themes are from here: https://github.com/vega/vega-themes/#vega-themes
def changing_themes(theme):
    with alt.themes.enable(theme):

        Figure06A= alt.Chart(Figure6A, title='a)').mark_line().encode(
        x= alt.X("Wavelength:Q", title="Wavelength / nm"),
        y=alt.Y("Flourescence_x10\u00b3:Q", title="Fluorescence (x10\u00b3) /a.u."),
        color=alt.Color('AminoAcidSideChains:N', title="Amino Acid Side Chains"),
        tooltip=['AminoAcidSideChains']
        ).transform_filter(
            alt.FieldOneOfPredicate(field='AminoAcidSideChains', oneOf=aminoAcids)
        )

        # Figure06A.show() # This command will block the Python interpreter until the browser window containing the chart is closed.
        Figure06A.display()
        # Note that calling .display() on multiple charts is the way that you can display multiple charts in a single cell.

# saving the DataFrame as a CSV file in the directory where this Jupyter Notebook exists
# the IF statement checks to see if we a csv file with that name exist in the directory
# The CSV file will often exist if you've run this notebook before or downloaded the files 
# but if not this is how you can create a csv file in the same directory

# if there is no .csv file with that name then we create a csv from the dataframe
# we'll know if it created on if the cell prints something

if not pathlib.Path('DigitalCommonsDesigningGraphsCSV01.csv').exists():
    csv_data = Figure6A.to_csv('DigitalCommonsDesigningGraphsCSV01.csv', index = False)
    print('\nCSV String:\n', csv_data)<jupyter_output><empty_output><jupyter_text>What we can add Next:a code example of creating a personalized 'lab' theme to then have it applied to the rest of the graphs for consistencyan example of Methods for including multiple graphs in the same plot and labeling them [Cool multi plot](https://stackoverflow.com/questions/59956588/altair-how-can-i-style-lines-differently-in-a-facet-grid-based-on-their-max-va)this returns an interesting graph that I felt might be a nice example for the people coming to the workshop     ---    import pydataset    from pydataset import data    df = data('sleepstudy')    alt.Chart(df).mark_line().encode(        x=alt.X('Days:O', title=''),        y=alt.Y('Reaction:Q', title='', scale=alt.Scale(domain=(200, 450))),         How do I test for the condition: is max(reaction) > 400 ms at least once during the study?        color=alt.condition(            alt.datum['Reaction'] > 300,            alt.value('orange'),            alt.value('darkgray')        ),        facet=alt.Facet('Subject:O', columns=3, title=''),    ).properties(        title='Reaction times (ms) for sleep-deprived subjects over 10 days',        width=200,        height=100    )    --- <NAME>,<NAME> sent me a lengthy reply today (see below). I don’t think that we need to present ChemDraw. It looks like she wants methods for creating plots that look consistent. She provides some examples at the end of this email. In summary she wants,•	How to change defaults on excel (I can handle this)•	Ways of creating templates that can make plots consistent •	Methods for including multiple graphs in the same plot and labeling them•	Show how to create a 3D plot•	Methods for managing really large datasets that slow the computer downWe can chat about these ideas at any time. I am relatively free tomorrow if that works for you. Figure design workshop plan- Issues, limitations of figure design    - Too many options for software    - Excel is too slow and requires a lot of tweaking- Principles of figure design    - Simplicity, color, marker differences    - Colorblind accommodations    - Consistency of plots- Tools    - Tableau    - MagicPlot Student    - R – ggplot2 or Python Francisco Begin making a folder in the Vanderbilt drive and start working on things like creating a powerpoint that address the first two bullets  Print DependencesDependences are fundamental to record the **computational environment**- I use [watermark](https://github.com/rasbt/watermark) to print version of python, ipython, packages, and characteristics of the computer then write a requirements.txt or environment.yml file    - [How to create a reproducible Jupyter notebook?](https://github.com/sbonaretti/cart_segm_liter_map)- There are other methods to automate creating a requirements.txt. For example, using the pipreqs to manages pip requirements files for multiple environments (e.g. production and development). Requires you to run command line    - [pipreqs](https://pypi.org/project/pipreq/:~:text=pipreq%20uses%20an%20rc%20file%20to%20track%20requirements.,be%20shared.%20The%20default%20configuration%20is%20as%20follows%3A)    - [pipreqsnb](https://pypi.org/project/pipreqsnb/) jupyter notebook wrapper for pipreqs **Note:** pipreqsnb is a bash application<jupyter_code>%load_ext watermark
%watermark
# author, last updated, python, ipython, modules, packages and machine characteristics
%watermark -a "<NAME>" 

print('Pythong version information')
%watermark -v

%watermark -u --iso8601 

print('Machine Information')
%watermark -m

# modules
print('Modules')
%watermark --iversions

# packages
print ("Packages")
%watermark -p watermark,pandas,altair,ipywidgets,IPython<jupyter_output>Author: <NAME>

Pythong version information
Python implementation: CPython
Python version       : 3.8.8
IPython version      : 7.22.0

Last updated: 2022-02-03T09:46:05.177150-06:00

Machine Information
Compiler    : MSC v.1916 64 bit (AMD64)
OS          : Windows
Release     : 10
Machine     : AMD64
Processor   : Intel64 Family 6 Model 142 Stepping 12, GenuineIntel
CPU cores   : 8
Architecture: 64bit

Modules
ipywidgets: 7.6.3
altair    : 4.1.0
pandas    : 1.3.5

Packages
watermark : 2.3.0
pandas    : 1.3.5
altair    : 4.1.0
ipywidgets: 7.6.3
IPython   : 7.22.0
{'fraction_non_alphanumeric': 0.08188865947251549, 'fraction_numerical': 0.017258171335073434, 'mean_word_length': 4.613382899628252, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 42, 'https://': 39, 'lorem ipsum': 0, 'www.': 7, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '272383', 'license': 'MIT', 'n_tokens_mistral': 7677, 'n_tokens_neox': 7097, 'n_words': 3014, 'path': 'Designing_Graphs_For_Publication_W_Altair.ipynb', 'repo_name': 'juarezfd/BinderTest', 'size': '25744'}