Next Article in Journal
Molecular Aspects of Varicella-Zoster Virus Latency
Previous Article in Journal
Protection against Virulent Infectious Bronchitis Virus Challenge Conferred by a Recombinant Baculovirus Co-Expressing S1 and N Proteins
Previous Article in Special Issue
AIDS Clinical Research in Spain—Large HIV Population, Geniality of Doctors, and Missing Opportunities
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Communication

Identification of Novel HIV-1 Latency-Reversing Agents from a Library of Marine Natural Products

by
Khumoekae Richard
1,†,
David E. Williams
2,†,
E. Dilip De Silva
3,
Mark A. Brockman
1,4,5,
Zabrina L. Brumme
1,5,
Raymond J. Andersen
2 and
Ian Tietjen
1,*
1
Faculty of Health Sciences, Simon Fraser University, Burnaby, BC V5A 1S6, Canada
2
Departments of Chemistry and Earth, Ocean & Atmospheric Sciences, University of British Columbia, Vancouver, BC V6T 1Z4, Canada
3
Department of Chemistry, University of Colombo, Colombo 03, Sri Lanka
4
Department of Molecular Biology and Biochemistry, Simon Fraser University, Burnaby, BC V5A 1S6, Canada
5
British Columbia Centre for Excellence in HIV/AIDS, Vancouver, BC V6Z 1Y6, Canada
*
Author to whom correspondence should be addressed.
Equal contribution.
Submission received: 13 June 2018 / Revised: 22 June 2018 / Accepted: 26 June 2018 / Published: 27 June 2018
(This article belongs to the Special Issue Homage to Mark Wainberg)

Abstract

:
Natural products originating from marine and plant materials are a rich source of chemical diversity and unique antimicrobials. Using an established in vitro model of HIV-1 latency, we screened 257 pure compounds from a marine natural product library and identified 4 (psammaplin A, aplysiatoxin, debromoaplysiatoxin, and previously-described alotaketal C) that induced expression of latent HIV-1 provirus in both cell line and primary cell models. Notably, aplysiatoxin induced similar levels of HIV-1 expression as prostratin but at up to 900-fold lower concentrations and without substantial effects on cell viability. Psammaplin A enhanced HIV-1 expression synergistically when treated in combination with the protein kinase C (PKC) activator prostratin, but not the histone deacetylase inhibitor (HDACi) panobinostat, suggesting that psammaplin A functions as a latency-reversing agent (LRA) of the HDACi class. Conversely, aplysiatoxin and debromoaplysiatoxin synergized with panobinostat but not prostratin, suggesting that they function as PKC activators. Our study identifies new compounds from previously untested marine natural products and adds to the repertoire of LRAs that can inform therapeutic “shock-and-kill”-based strategies to eliminate latent HIV-infected reservoirs.

Graphical Abstract

While current licensed HIV-1 therapies inhibit virus replication, they do not act on latently-infected CD4+ T cells which have HIV-1 provirus incorporated within their genomes. As these proviruses can reactivate at any time to produce infectious virus, novel approaches are needed to eliminate these HIV-1 reservoirs [1,2,3]. One therapeutic strategy, frequently termed “shock-and-kill” [4], proposes the treatment of latent HIV-infected cells with latency-reversing agents (LRAs) to induce proviral expression (“shock”), after which these cells may be eliminated by viral cytopathic effects or host immune responses (“kill”). Numerous LRAs have been identified that belong to distinct functional classes, primarily histone deacetylase inhibitors (HDACi) and protein kinase C (PKC) activators [5,6]. However, while several of these LRAs reproducibly induce HIV-1 proviral expression in vitro and/or in vivo, to date no LRA has appreciably reduced the size of the inducible viral reservoir in clinical trials, underscoring a need for new LRAs with improved efficacy [5,6,7]. Toward this goal, several studies show that treatment of cells with combinations of LRAs from distinct functional classes yields synergistic responses that are significantly greater than the expected additive effects of these agents [8,9,10,11,12,13,14,15]. However, as outcomes from clinical trials with LRA combinations are not yet reported, continued discovery of additional LRAs and LRA combinations remains a priority.
Natural products obtained from plant and marine sources are a rich source of diverse chemical compounds, including HIV-1 inhibitors and novel LRAs [16,17]. Moreover, screens for novel LRAs from natural product libraries often result in “hit” rates of 1.0% or more [18,19], indicating that even small natural chemical libraries may be sufficient to identify new agents of interest. Here, we describe the results from a screen for novel LRAs from a library of 257 pure and structurally diverse natural compounds derived from marine invertebrates and microorganisms. These compounds were assembled over many years by the laboratory of R.J.A. (e.g., [20,21,22,23,24]); however, many of these compounds have no known molecular targets or have only been reported to possess basic cytotoxic or antimicrobial activity. We previously screened this library for compounds that inhibit HIV replication and identified 6 with 50% effective concentrations (EC50s) of 3.8 µM or less, including at least one (bengamide A) that acts by mechanisms which are distinct from licensed antiretrovirals [25]. Here we describe 4 new LRAs identified from this library, including 1 HDACi and 3 PKC activators.
All natural compounds were confirmed ≥ 95% pure by NMR and LC/MS and dissolved in DMSO to stock concentrations of 5 mg/mL. The control LRA TNFα (Sigma-Aldrich, Oakville, ON, Canada) was dissolved in PBS plus 0.1% bovine serum albumin to a stock concentration of 5 µg/mL. The control LRA panobinostat (HDACi; Sigma-Aldrich) was dissolved in DMSO to a stock concentration of 30 mM. The control LRA prostratin (PKC activator; Selleck Chemicals, Houston, TX, USA) was dissolved in PBS to a stock concentration of 1.5 mM.
J-Lat Full Length T cell lines (clones 8.4, 9.2, and 10.6) were obtained from the NIH AIDS Reagent Program, Division of AIDS, NIAID, NIH (contributed by Dr. Eric Verdin) [26]. These cells are derived from Jurkat cells and contain a transcriptionally-silent, HIV-1 proviral genome encoding a frameshift mutation within Env, and where Nef is replaced with a GFP reporter [26]. As a result, GFP expression in these cells indicates reactivation of HIV-1 from latency. Cells were maintained in R10+ medium (RPMI 1640 with HEPES and L-Glutamine (Lonza, Mississauga, ON, Canada), 10% fetal calf serum, 100 U/mL penicillin, and 100 µg/mL streptomycin (Sigma-Aldrich)) at 37 °C and 5% CO2.
For latency-reversal assays, J-Lat cells were re-suspended in R10+ medium to a concentration of 106 cells/mL. 2 × 105 cells were then seeded into 96-well plate wells with compounds or control LRAs at desired final concentrations. Control cells were incubated at a final concentration of up to 0.3% DMSO vehicle control. In no case did final test concentrations of DMSO exceed 0.3%, which had no observable effects on cell viability or GFP expression. Cells were incubated at 37 °C and 5% CO2 for 24 h, after which a minimum of 5000 cells from each culture was analyzed by flow cytometry (Guava EasyCyte 8HT, EMD Millipore, Etobicoke, ON, Canada). Flow cytometry data were analyzed using FlowJo v. 8.8.7 software (FlowJo LLC, Ashland, OR, USA), and results were reported as the mean ± s.e.m. from at least 3 independent experiments.
Compounds were initially assessed in J-Lat 9.2 cell cultures based on observations from us and others that this cell line responds to LRAs from multiple functional classes but rarely achieves GFP expression in all cells, thereby enabling comparative dose-response studies across large concentration ranges in addition to accurate studies of synergistic effects of LRA combinations [12,14,15]. Representative flow cytometry data using this cell line and control LRAs are shown in Figure 1A. For each experiment, control J-Lat cell cultures treated with 0.1% DMSO were first gated for typical forward and side-scatter profiles consistent with viable, healthy cells [14,15]. This subset was then assessed for GFP expression across all cell cultures, with the GFP-positive gate set such that the average of GFP-positive cells in DMSO-treated control cell cultures was 0.05% (Figure 1A, top). For each culture treated with compound, the extent of latency reversal was then measured as the percentage of GFP-positive cells. For example, in one representative experiment, treatment of J-Lat cells with 0.1 µg/mL panobinostat (~0.15 µM) resulted in 13.9% GFP-positive cells, while treatment with 10 µg/mL prostratin (~12 µM) resulted in 6.2% GFP-positive cells (Figure 1A, center and bottom). Consistent with published data [8,15,27], both control LRAs exhibited dose-dependent expression of GFP across multiple concentrations (Figure 1B). For example, maximal GFP expression in J-Lat cells was observed after treatment with 1.4 µM panobinostat (17.1 ± 2.7%) or 12 µM prostratin (6.4 ± 0.9%) (Figure 1B). Using the approach of Hashemi et al. [28] and normalizing results relative to the average GFP response for 12 µM prostratin (the concentration at which maximal prostratin activity was observed), the EC50s for panobinostat and prostratin were calculated to be 0.10 ± 0.02 and 7.1 ± 2.8 µM, respectively (Table 1). These results confirm that panobinostat is approximately 70-fold more potent than prostratin, which is also consistent with previous reports [8,15,27].
To directly assess the impact of control LRAs on cell viability (i.e., in a manner independent of provirus expression), parental Jurkat cells (Clone E6-1, American Type Culture Collection; Manassas, VA, USA), which do not harbor integrated HIV-1 provirus, were prepared and treated with LRAs as described above. Following 24 h incubation, compound toxicity was assessed by measuring surface expression of the early apoptotic marker annexin V by flow cytometry (by staining with annexin V-APC; BioLegend, San Diego, CA, USA). Results were reported as the fold-increase in annexin V-positive cells relative to 0.1% DMSO-treated control cells (mean ± s.e.m.) from at least 3 independent experiments. Representative data are shown in Figure 1C. Here, treatment with 0.1% DMSO resulted in 13.6% of cells with APC-fluorescence above the bulk of the cell population, which is presumed to lack surface expression of annexin V (top). In contrast, treatment with 0.15 µM panobinostat resulted in 58.6% APC-positive cells (i.e., a 4.3-fold increase in apoptosis from DMSO control) (center), while 12 µM prostratin resulted in 24.5% APC-positive cells (i.e., 1.8-fold increase from DMSO control) (bottom). As expected, control LRAs increased cellular apoptosis in a dose-dependent manner (Figure 1D). For example, treatment with 0.045 µM panobinostat induced a 5.8 ± 1.2-fold increase in apoptosis, indicating poor cellular tolerance at concentrations that induced latency reversal, while at least 10 µM prostratin induced no more than a 2.0 ± 0.3-fold increase in apoptosis (Figure 1D). Taken together, our control experiments confirm that panobinostat is a more potent, yet also more toxic, LRA than prostratin [8,15,27].
We next screened 257 structurally-diverse pure compounds derived from marine natural products at 2.5 µg/mL for latency reversal activity in J-Lat 9.2 cells. Of these, seven (2.7%) compounds resulted in cytolysis and disruption of cell morphology as observed by light microscopy, consistent with widespread cell death, and were not considered further. Of the remaining 250 compounds, four induced GFP expression in at least 4% of cells (Figure 2A). This “hit” rate of 1.6% is in line with previously reported screens of natural product libraries (~0.5–1.1%) [18,19] and supports the notion that pure natural product libraries are enriched for bioactive LRAs compared to synthetic small molecule libraries, where reported hit rates of ~0.1% are more frequent [28,35,36].
Structures of the four identified compounds are shown in Figure 2B. Notably, psammaplin A, originally isolated from the two-sponge associate Poecillastra sp. and Jaspis sp. [29], was previously identified as an HDACi with anti-tumor activity [30,31]. In contrast, aplysiatoxin and debromoaplysiatoxin, which differs from aplysiatoxin by the loss of a bromine atom in the phenol ring, are toxins produced by blue-green algae and potent PKC activators [32,33,34]. However, not all HDACis and PKC activators possess potent HIV latency modulatory functions [14,37,38], and none of these compounds have been investigated as HIV-1 LRAs. Finally, alotaketal C, originally isolated from Phorbas sp., is a potent activator of cyclic AMP and PKC signaling that we recently characterized for its HIV-1 latency reversal activity [14,20] and is thus not assessed further here.
Each new LRA induced dose-dependent reversal of HIV-1 latency in J-Lat cells across multiple concentrations (Figure 2C,D). The average maximum responses observed for each compound ranged from 1.4 to 2.7-fold more than controls treated with 12 µM prostratin. For example, psammaplin A induced GFP expression in up to 17.6 ± 4.0% of cells at 5 µg/mL (3.8 µM). When results were normalized to the average GFP response for 12 µM prostratin [28], psammaplin A’s EC50 was calculated as 1.9 ± 0.3 µM, approximately 19-fold higher than the EC50 of panobinostat (Table 1). In contrast, aplysiatoxin induced GFP expression in 9.4 ± 0.1% cells with as little as 0.15 µg/mL (0.1 µM) and yielded a calculated EC50 of 0.045 ± 0.021 µM. Aplysiatoxin is therefore 160-fold more potent than prostratin and 2.2-fold more than panobinostat, identifying it as a particularly potent LRA in J-Lat 9.2 cells. Debromoaplysiatoxin induced GFP expression in 7.2 ± 1.5% of cells at 1.5 µg/mL (1.3 µM) and yielded a calculated EC50 of 0.92 ± 0.14 µM, or 7.7-fold more potent than prostratin.
In Jurkat cells, we observed that 3.8 µM psammaplin A induced a 3.3 ± 0.6-fold increase in annexin V staining, with extensive cell death observed at higher concentrations. In contrast, no more than 2.0 ± 0.2 and 1.6 ± 0.1-fold increases in annexin V staining were observed for aplysiatoxin and debromoaplysiatoxin, respectively (Figure 2E,F). Thus, psammaplin A appeared to be toxic at concentrations that induced latency reversal, while both aplysiatoxin and debromoaplysiatoxin were largely well-tolerated across all concentrations.
These agents displayed similar dose-response profiles in J-Lat 8.4 cells, indicating that they act on HIV provirus independent of its integration site (Figure 3A). For example, when results were normalized to the average GFP response for 38 µM prostratin (i.e., the concentration at which maximal activity was observed in J-Lat 8.4 cells), the EC50 of psammaplin A was calculated as 1.5 ± 0.1 µM, or approximately 20.5-fold higher than the EC50 of panobinostat (Table 1). Aplysiatoxin induced detectable GFP expression at concentrations as low as 0.0001 µM, with a calculated EC50 of 0.011 ± 0.003 µM, or 900-fold more potent than prostratin and 6.6-fold more than panobinostat. Similarly, an EC50 of 0.52 ± 0.02 µM was calculated for debromoaplysiatoxin, or 19.2-fold more potent than prostratin.
In J-Lat 10.6 cells, we observed that all LRAs were capable of inducing GFP expression in at least two-thirds of cells (Figure 3B), indicating robust efficacy. However, we also observed an average of 7.5% of J-Lat 10.6 cells spontaneously expressing GFP in the absence of LRAs, consistent with previous reports [39], indicating a lower barrier to HIV latency reversal compared to J-Lat 9.2 and 8.4. Nevertheless, when results were normalized to the average GFP response for 3.8 µM prostratin (one of three concentrations where maximum activity was observed), the EC50 of psammaplin A was again calculated as 1.5 ± 0.1 µM, or approximately 36.6-fold higher than the EC50 of panobinostat (Table 1). Moreover, GFP expression was observed with aplysiatoxin concentrations as low as 3.7 × 10−5 µM (37 pM) and a calculated EC50 of 0.0033 ± 0.0012 µM, or 540-fold more potent than prostratin and 12.4-fold more than panobinostat. Finally, the calculated EC50 of debromoaplysiatoxin (0.081 ± 0.029 µM) was 22.2-fold more potent than prostratin. Thus, the rank-order of potency for all LRAs was consistent across all cell lines.
To confirm that LRAs induce HIV protein expression in addition to the GFP reporter, J-Lat 10.6 cells were also stained with the HIV-1 p24Gag antibody KC57-RD1 (Beckman Coulter, Indianapolis, IN, USA) and processed using the Cytofix/Cytoperm Fixation/Permeabilization Kit (BD Biosciences, Mississauga, ON, Canada) prior to flow cytometric analysis. Results were then reported as the fold-increase in p24Gag-positive cells relative to 0.1% DMSO-treated control cells (mean ± s.e.m.) from at least 3 independent experiments. All LRAs were observed to induce at least 9.7-fold increased p24Gag-positive cells, with the same rank order as observed for GFP expression (Figure 3C). This confirms that LRAs also induce viral protein expression.
To investigate whether LRAs induce proviral expression in primary human cells, we obtained peripheral blood mononuclear cells (PBMCs) from three HIV-infected donors on stably-suppressive antiretroviral therapy for at least three years (Figure 4). Study protocols were approved by the Institutional Review Boards of Simon Fraser University and the University of British Columbia—Providence Health Care Research Institute (REB: H15-03077, approved 8 March 2016). Written informed consent was obtained from all donors. Frozen PBMC aliquots were thawed and allowed to recover in R10+ medium at 37 °C, 5% CO2 for 24 h at 2.5 × 106 cells/mL. PBMCs were then incubated at 106 cells/mL with positive control cell activators PMA (100 ng/mL) plus ionomycin (1 µg/mL), 3.8 µM psammaplin A, 1.1 µM aplysiatoxin, 1.3 µM debromoaplysiatoxin, or 0.1% DMSO vehicle control. All conditions were performed in duplicate. Following 24 h incubation at 37 °C and 5% CO2, supernatant p24Gag protein was quantified by ELISA (Xpress Bio, Frederick, MD, USA), and cell viability was measured by flow cytometry using Guava Viacount, a DNA intercalating dye (EMD Millipore). In most cases, each LRA caused an increase in supernatant p24Gag above background; however, substantial donor-to-donor variation was observed, consistent with other studies [10,11,12] (Figure 4A). For example, while treatment with PMA + ionomycin induced an average 62.3 ± 31.1% (mean ± s.e.m.) increase in supernatant p24Gag relative to untreated cells, psammaplin A resulted in a 85.0 ± 41.6% increase. Similarly, aplysiatoxin and debromoaplysiatoxin induced increases of 56.0 ± 15.0 and 46.9 ± 22.3%, respectively. No major changes in cell viability were observed, with a maximum 15.4 ± 2.2% (mean ± s.e.m.) reduction in viability in the presence of 3.8 µM psammaplin A (Figure 4B). These results indicate that LRAs have the capacity to activate latent HIV-1 provirus in primary human cells isolated from persons with long-term viremia suppression on antiretroviral therapy.
As described previously [8,9,10,11,12,13,14,15], treatment of cells with combinations of LRAs acting through different mechanisms tends to result in synergistic effects on HIV-1 latency reversal, while treatment with compounds acting through similar mechanisms tends to yield at most additive responses. These observations can therefore be used to identify potential functional classes of novel LRAs [14,15]. To demonstrate this, we assessed GFP expression in J-Lat 9.2 cells treated with novel LRAs in combination with control LRAs, including the pro-inflammatory cytokine TNFα, the HDACi panobinostat, and the PKC activator prostratin (Figure 5). In these studies, synergism was observed in all cases where control LRAs were applied in combination: for example, treatment of J-Lat 9.2 cells separately with 0.01 µg/mL TNFα or 0.15 µM panobinostat induced 22.3 ± 2.5% and 8.9 ± 1.8% GFP-positive cells, respectively, whereas treatment of cells with both compounds simultaneously led to 49.0 ± 1.2% GFP-positive cells, which is ~1.6-fold higher than would be expected if the effects of these two compounds were strictly additive (i.e., 31.2%; Figure 5). Similarly, treatment with TNFα plus 12 µM prostratin, or panobinostat plus prostratin, induced responses that were 1.6- and 2.6-fold higher than expected for additive effects, respectively. These levels of synergism were statistically significant (p < 0.05; Student’s unpaired t-test with a two-sided, Bonferroni correction) as measured by the Bliss independence model, which was calculated as described previously [10,11].
Treatment of J-Lat 9.2 cells with 2 µM psammaplin A in addition to TNFα or prostratin induced 49.8 ± 1.1 and 44.4 ± 5.1% GFP-positive cells, representing 1.7- and 2.7-fold increases in GFP expression over what would be expected from strictly additive effects, respectively. However, treatment of J-Lat cells with psammaplin A plus panobinostat induced only 9.8 ± 1.7% GFP-positive cells, or 0.6-fold of what would be expected by strictly additive effects (Figure 5). These results are consistent with the known function of psammaplin A as a HDACi [30,31]. Conversely, treatment of cells with 2 µM aplysiatoxin plus TNFα or panobinostat induced 51.1 ± 0.5 and 48.4 ± 2.2% GFP-positive cells, respectively, or 1.5- and 2.3-fold increases in GFP-positive cells over expected additive effects, while co-treatment with prostratin induced only 12.2 ± 1.8% GFP-positive cells, or 0.6-fold of expected additive effects. These observations are consistent with the known function of aplysiatoxin as a PKC activator [33,34]. Similar results were found using debromoaplysiatoxin: co-treatment with TNFα or panobinostat induced 50.4 ± 1.5 and 47.6 ± 1.5% GFP-positive cells, respectively, or 1.5 or 2.3-fold over expected additive effects, while co-treatment with prostratin induced 12.0 ± 1.8% GFP-positive cells, or 0.6-fold of expected additive effects, indicating that its latency reversal activity is also likely due to activation of PKC. All synergistic effects were statistically significant as measured with the Bliss independence model. Taken together, these results indicate that the latency reversal properties of these pure natural products in J-Lat cells are consistent with their previously reported functional properties.
In summary, we identify four LRAs derived from marine natural products that can be added to the repertoire of known HIV-1 shock-and-kill agents. The likely mechanisms of action for all four compounds, supported here and by prior studies, are consistent with established functional classes, including one HDACi (psammaplin A) and three PKC activators (aplysiatoxin, debromoaplysiatoxin, and previously-described alotaketal C) [14]. Dose response profiles suggest that psammaplin A is a less potent LRA than panobinostat, while aplysiatoxin and debromoaplysiatoxin are more potent than prostratin. These observations were confirmed in two additional J-Lat cell lines and in PBMCs from three HIV-infected donors, indicating that latency reversal occurs independent of proviral integration sites and that these compounds can reverse latency in primary cells. The contributions of these mechanisms to HIV-1 latency reversal were further supported by synergism studies, described both here and elsewhere [14].
Our previous discovery of multiple novel HIV inhibitors from this chemically-diverse library (e.g., [25]) led us to hypothesize that we might also identify new modulators of HIV latency with distinct molecular mechanisms. However, while this screen instead identified only compounds of the HDACi and PKC activation classes, the results described here should not preclude future screening of natural product-derived compound libraries for additional LRAs which may act by novel mechanisms of action. Conversely, as the kinetic properties of both HDACis and PKC activators do not necessarily correspond with latency reversal efficacy [14,37,38], testing of library compounds with both known and unknown molecular functions remains warranted. In support of this, we notably identify one PKC activator, aplysiatoxin, that is particularly potent across multiple cell models, with activity observed in J-Lat 8.4 and 10.6 cells at picomolar concentrations. Finally, we note that additional latency reversal mechanisms by these agents may remain undetected. For example, we previously described alotaketal C as an LRA of the PKC activator class [14], but it is additionally reported to function as an activator of cyclic AMP signaling [20], which may also modulate HIV-1 latency reversal [40,41].
Taken together, this study highlights the benefits of natural product-based screens for LRA discovery. The identification and evaluation of new LRAs will support the development of novel therapeutic combinations and clinical approaches to reduce or eliminate latent HIV-1 reservoirs.

Author Contributions

K.R., M.A.B., Z.L.B., and I.T. designed and/or performed experiments. D.E.W., E.D. S., and R.J.A. isolated and characterized natural product compounds. All authors contributed to writing of the manuscript.

Acknowledgments

We thank the donors for PBMC samples and Silven Read for superb technical assistance. Funding was provided by the Canadian Institutes for Health Research (CIHR PJT-153057) (I.T., M.A.B., Z.L.B.), the Canadian Foundation for AIDS Research (CANFAR) (I.T., M.A.B., Z.L.B.), a Simon Fraser University President’s Research Start-up Grant (I.T.), the Canadian HIV Cure Enterprise Team Grant (HIG-133050) (M.A.B., Z.L.B.) from the CIHR in partnership with CANFAR and the International AIDS Society, and the Natural Sciences and Engineering Council of Canada (R.J.A.). This work was also supported through the Sub-Saharan African Network for TB/HIV Research Excellence (SANTHE) (K.R., I.T.), a DELTAS Africa Initiative [grant # DEL-15-006]. The DELTAS Africa Initiative is an independent funding scheme of the African Academy of Sciences (AAS)’s Alliance for Accelerating Excellence in Science in Africa (AESA) and supported by the New Partnership for Africa’s Development Planning and Coordinating Agency (NEPAD Agency) with funding from the Wellcome Trust [grant # 107752/Z/15/Z] and the UK government. The views expressed in this publication are those of the author(s) and not necessarily those of AAS, NEPAD Agency, Wellcome Trust, or the UK government. K.R. is the recipient of a Canadian Queen Elizabeth II Diamond Jubilee Scholarship, a partnership between the Rideau Hall Foundation, Community Foundations of Canada and Universities Canada. M.A.B. holds a Tier 2 Canada Research Chair in viral pathogenesis and immunity. Z.L.B. is supported by a Scholar Award from the Michael Smith Foundation for Health Research.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Cary, D.C.; Fujinaga, K.; Peterlin, B.M. Molecular Mechanisms of HIV latency. J. Clin. Investig. 2016, 126, 448–454. [Google Scholar] [CrossRef] [PubMed]
  2. Finzi, D.; Hermankova, M.; Pierson, T.; Carruth, L.M.; Buck, C.; Chaisson, R.E.; Quinn, T.C.; Chadwick, K.; Margolick, J.; Brookmeyer, R.; et al. Identification of a reservoir for HIV-1 in patients on highly active antiretroviral therapy. Science 1997, 278, 1295–1300. [Google Scholar] [CrossRef] [PubMed]
  3. Siliciano, J.D.; Kajdas, J.; Finzi, D.; Quinn, T.C.; Chadwick, K.; Margolick, J.B.; Kovacs, C.; Gange, S.J.; Siliciano, R.F. Long-term follow-up studies confirm the stability of the latent reservoir for HIV-1 in resting CD4+ T cells. Nat. Med. 2003, 9, 727–728. [Google Scholar] [CrossRef] [PubMed]
  4. Deeks, S.G. HIV: Shock and kill. Nature 2012, 487, 439–440. [Google Scholar] [CrossRef] [PubMed]
  5. Archin, N.M.; Margolis, D.M. Emerging strategies to deplete the HIV reservoir. Curr. Opin. Infect. Dis. 2014, 27, 29–35. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  6. Spivak, A.M.; Planelles, V. Novel latency reversal agents for HIV-1 cure. Annu. Rev. Med. 2017. [Google Scholar] [CrossRef] [PubMed]
  7. Rasmussen, T.A.; Lewin, S.R. Shocking HIV out of hiding: Where are with with clinical trials of latency reversing agents? Curr. Opin. HIV AIDS 2016, 11, 394–401. [Google Scholar] [CrossRef] [PubMed]
  8. Reuse, S.; Calao, M.; Kabeya, K.; Guiguen, A.; Gatot, J.S.; Quivy, V.; Vanhulle, C.; Lamine, A.; Vaira, D.; Demonte, D.; et al. Synertistic activation of HIV-1 expression by deacetylase inhibitors and prostratin: Implications for treatment of latent infection. PLoS ONE 2009, 4, e6093. [Google Scholar] [CrossRef] [PubMed]
  9. Pérez, M.; de Vinuesa, A.G.; Sanchez-Duffhues, G.; Marquez, N.; Bellido, M.L.; Muñoz-Fernández, M.Á.; Moreno, S.; Castor, T.P.; Calzado, M.A.; Muñoz, E. Bryostatin-1 synergizes with histone deacteylase inhibitors to reactivate HIV-1 from latency. Curr. HIV Res. 2010, 8, 418–429. [Google Scholar] [CrossRef] [PubMed]
  10. Laird, G.M.; Bullen, C.K.; Rosenbloom, D.I.; Martin, A.R.; Hill, A.L.; Durand, C.M.; Siliciano, J.D.; Siliciano, R.F. Ex vivo analysis identifies effective HIV-1 latency-reversing drug combinations. J. Clin. Investig. 2015, 125, 1901–1912. [Google Scholar] [CrossRef] [PubMed]
  11. Jiang, G.; Mendes, E.A.; Kaiser, P.; Wong, D.P.; Tang, Y.; Cai, I.; Fenton, A.; Melcher, G.P.; Hildreth, J.E.; Thompson, G.R.; et al. Synergistic reactivation of latent HIV expression by ingenol-3-angelate, PEP005, targeted NF-κB signaling in combination with JQ1 induced p-TEFB activation. PLoS Pathog. 2015, 11, e1005066. [Google Scholar] [CrossRef] [PubMed]
  12. Darcis, G.; Kula, A.; Bouchat, S.; Fujinaga, K.; Corazza, F.; Ait-Ammar, A.; Delacourt, N.; Melard, A.; Kabeya, K.; Vanhulle, C.; et al. An in-depth comparison of latency-reversing agent combinations in various in vitro and ex vivo HIV-1 latency models identified bryostatin-1+ JQ1 and ingenol-B+ JQ1 to potently reactivate viral gene expression. PLoS Pathog. 2015, 11, e1005063. [Google Scholar] [CrossRef] [PubMed]
  13. Martinez-Bonet, M.; Clemente, M.I.; Serramía, M.J.; Muñoz, E.; Moreno, S.; Muñoz-Fernández, M.Á. Synergistic activation of latent HIV-1 expression by novel histone deacetylase inhibitors and bryostatin-1. Sci. Rep. 2015, 5, 16445. [Google Scholar] [CrossRef] [PubMed]
  14. Wang, M.; Tietjen, I.; Chen, M.; Williams, D.E.; Daoust, J.; Brockman, M.A.; Andersen, R.J. Sesterterpenoids isolated from the sponge Phorbas sp. activate latent HIV-1 provirus expression. J. Org. Chem. 2016, 81, 11324–11334. [Google Scholar] [CrossRef] [PubMed]
  15. Tietjen, I.; Ngwenya, B.N.; Fotso, G.; Williams, D.E.; Simonambango, S.; Ngadjui, B.T.; Andersen, R.J.; Brockman, M.A.; Brumme, Z.L.; Andrae-Marobela, K. The Croton megalobotrys Müll Arg. traditional medicine in HIV/AIDS management: Documentation of patient use, in vitro activation of latent HIV-1 provirus, and isolation of active phorbol esters. J. Ethnopharmacol. 2017. [Google Scholar] [CrossRef] [PubMed]
  16. Zhou, X.; Liu, J.; Yang, B.; Lin, X.; Yang, X.W.; Liu, Y. Marine natural products with anti-HIV activities in the last decade. Curr. Med. Chem. 2013, 20, 953–973. [Google Scholar] [PubMed]
  17. Cary, D.C.; Peterlin, D.M. Natural products and HIV/AIDS. AIDS Res. Hum. Retrovir. 2018, 34, 31–38. [Google Scholar] [CrossRef] [PubMed]
  18. Yang, H.C.; Xing, S.; Shan, L.; O’Connell, K.; Dinoso, J.; Shen, A.; Zhou, Y.; Shrum, C.K.; Han, Y.; Liu, J.O.; et al. Small-molecule screening using a human primary cell model of HIV latency identifies compounds that reverse latency without cellular activation. J. Clin. Investig. 2009, 119, 3473–3485. [Google Scholar] [CrossRef] [PubMed]
  19. Doyon, G.; Sobolewski, M.D.; Huber, K.; McMahon, D.; Mellors, J.W.; Sluis-Cremer, N. Discovery of a small molecule agonist of phosphatidylinositol 3-kinase p110α that reactivates latent HIV-1. PLoS ONE 2014, 9, e84964. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  20. Daoust, J.; Chen, M.; Wang, M.; Williams, D.E.; Garcia Chavez, M.A.; Yang, Y.A.; Merchant, C.E.; Fontana, A.; Kieffer, T.J.; Andersen, R.J. Sesterterpenoids isolated from a northeaster Pacific Phorbas sp. J. Org. Chem. 2013, 78, 8267–8273. [Google Scholar] [CrossRef] [PubMed]
  21. Li, D.; Carr, G.; Zhang, Y.; Williams, D.E.; Amlani, A.; Bottriell, H.; Mui, A.L.-F.; Andersen, R.J. Turnagainolides A and B, cyclic depsipeptides produced in culture by a Bacillus sp.: Isolation, structure elucidation, and synthesis. J. Nat. Prod. 2011, 74, 1093–1099. [Google Scholar] [CrossRef] [PubMed]
  22. Williams, D.E.; Dalisay, D.S.; Patrick, B.O.; Matainaho, T.; Andrusiak, K.; Deshpande, R.; Myers, C.L.; Piotrowski, J.S.; Boone, C.; Yoshida, M.; et al. Padanamides A and B, highly modified linear tetrapeptides produced in culture by a Streptomyces sp. isolated from marine sediment. Org. Lett. 2011, 13, 3936–3939. [Google Scholar] [CrossRef] [PubMed]
  23. Williams, D.E.; Steinø, A.; de Voogd, N.J.; Mauk, A.G.; Andersen, R.J. Halicloic acids A and B isolated from the marine sponge Haliclona sp. collected in the Phillipines inhibit indoleamine 2,3-dioxygenase (IDO). J. Nat. Prod. 2012, 75, 1451–1458. [Google Scholar] [CrossRef] [PubMed]
  24. Williams, D.E.; Dalisay, D.S.; Li, F.; Amphlett, J.; Maneerat, W.; Matainaho, T.; Yu, W.; Brown, P.J.; Arrowsmith, C.H.; Vedadi, M.; et al. Nahuoic acid A produced by a Streptomyces sp. isolated from a marine sediment is a selective SAM-competitive inhibitor of the histone methyltransferase SETD8. Org. Lett. 2013, 15, 414–417. [Google Scholar] [CrossRef] [PubMed]
  25. Tietjen, I.; Williams, D.E.; Read, S.; Kuang, X.T.; Mwimanzi, P.; Wilhelm, E.; Markle, T.; Kinloch, N.N.; Naphen, C.N.; Tenney, K.; et al. Inhibition of NF-κB-dependent HIV-1 replication by the marine natural product bengamide A. Antivir. Res. 2018, 152, 94–103. [Google Scholar] [CrossRef] [PubMed]
  26. Jordan, A.; Bisgrove, D.; Verdin, E. HIV reproducibly establishes a latent infection after acute infection of T cells in vitro. EMBO J. 2003, 22, 1868–1877. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  27. Spina, C.A.; Anderson, J.; Archin, N.M.; Bosque, A.; Chan, J.; Famiglietti, M.; Greene, W.C.; Kashuba, A.; Lewin, S.R.; Margolis, D.M.; et al. An in-depth comparison of latent HIV-1 reactivation in multiple cell models systems and resting CD4+ T cells from aviremic patients. PLoS Pathog. 2013, 9, e1003834. [Google Scholar] [CrossRef] [PubMed]
  28. Hashemi, P.; Barreto, K.; Bernhard, W.; Lomness, A.; Nonson, N.; Pfeifer, T.A.; Harrigan, P.R.; Sadowski, I. Compounds producing an effective combinatorial regimen for disruption of HIV-1 latency. EMBO Mol. Med. 2017. [Google Scholar] [CrossRef] [PubMed]
  29. Jung, J.H.; Sim, C.J.; Lee, C.O. Cytotoxic compounds from a two-sponge association. J. Nat. Prod. 1995, 58, 1722–1726. [Google Scholar] [CrossRef] [PubMed]
  30. Kim, D.H.; Shin, J.; Kwon, H.J. Psammaplin A is a natural prodrug that inhibits class I histone deacetylase. Exp. Mol. Med. 2007, 39, 47–55. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  31. Baud, M.G.; Leiser, T.; Haus, P.; Samlal, S.; Wong, A.C.; Wood, R.J.; Petrucci, V.; Gunaratnam, M.; Hughes, S.M.; Buluwela, L.; et al. Defining the mechanism of action and enzymatic selectivity of psammaplin A against its epigenetic targets. J. Med. Chem. 2012, 55, 1731–1750. [Google Scholar] [CrossRef] [PubMed]
  32. Nagai, H.; Yasumoto, T.; Hokama, Y. Aplysiatoxin and debromoaplysiatoxin as the causative agents of a red alga Gracilaria coronopifolia poisoning in Hawaii. Toxicon 1996, 34, 753–761. [Google Scholar] [CrossRef]
  33. Arcoleo, J.P.; Weinstein, I.B. Activation of protein kinase C by tumor promoting phorbol esters, teleocidin and aplysiatoxin in the absence of added calcium. Carcinogenesis 1985, 6, 213–217. [Google Scholar] [CrossRef] [PubMed]
  34. Nakamura, H.; Kishi, Y.; Pajares, M.A.; Rando, R.R. Structural basis of protein kinase C activation by tumor promoters. Proc. Natl. Acad. Sci. USA 1989, 86, 9672–9676. [Google Scholar] [CrossRef] [PubMed]
  35. Micheva-Viteva, S.; Kobayashi, Y.; Edelstein, L.C.; Pacchia, A.L.; Lee, H.L.R.; Graci, J.D.; Breslin, J.; Phelan, B.D.; Miller, L.K.; Colacino, J.M.; et al. High-throughput screening uncovers a compound that activates latent HIV-1 and acts cooperatively with a histone deacetylase (HDAC) inhibitor. J. Biol. Chem. 2011, 286, 21083–21091. [Google Scholar] [CrossRef] [PubMed]
  36. Gallastegui, E.; Marshall, B.; Vidal, D.; Sanchez-Duffhues, G.; Collado, J.A.; Alvarez-Fernández, C.; Luque, N.; Terme, J.M.; Gatell, J.M.; Sánchez-Palomino, S.; et al. Combination of biological screening in a cellular model of viral latency and virtual screening identifies novel compounds that reactivate HIV-1. J. Virol. 2012, 86, 3795–3808. [Google Scholar] [CrossRef] [PubMed]
  37. Shan, L.; Xing, S.; Yang, H.-C.; Zhang, H.; Margolick, J.B.; Siliciano, R.F. Unique characteristics of histone deacetylase inhibitors in reactivation of latent HIV-1 in Bcl-2-transduced primary resting CD4+ T cells. J. Antimicrob. Chemother. 2014, 69, 28–33. [Google Scholar] [CrossRef] [PubMed]
  38. Beans, E.J.; Fournogerakis, D.; Gauntlett, C.; Heumann, L.V.; Kramer, R.; Marsden, M.D.; Murray, D.; Chun, T.-W.; Zack, J.A.; Wender, P.A. Highly potent, synthetically accessible prostratin analogs induce latent HIV expression in vitro and ex vivo. Proc. Natl. Acad. Sci. USA 2013, 110, 11698–11703. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  39. Romani, B.; Jamil, R.K.; Hamidi-Fard, M.; Rahimi, P.; Momen, S.B.; Aghasadeghi, M.R.; Allahbakhshi, E. HIV-1 Vpr reactivates latent HIV-1 provirus by inducing depletion of class I HDACs on chromatin. Sci. Rep. 2016, 6, 31924. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  40. Rabbi, M.F.; Al-Harthi, L.; Saifuddin, M.; Roebuck, K.A. The cAMP-dependent protein kinase A and protein kinase C-beta pathways synergistically interact to activate HIV-1 transcription in latently infected cells of monocyte/macrophage lineage. Virology 1998, 245, 257–269. [Google Scholar] [CrossRef] [PubMed]
  41. Lim, H.; Kim, K.C.; Son, J.; Shin, Y.; Yoon, C.H.; Kang, C.; Choi, B.S. Synergistic reactivation of latent HIV-1 provirus by PKA activator dibutryl-cAMP in combination with an HDAC inhibitor. Virus Res. 2017, 227, 1–5. [Google Scholar] [CrossRef] [PubMed]
Figure 1. HIV-1 latency reversal and cellular toxicity assessments of control LRAs. (A) Representative GFP expression in J-Lat cells treated with 0.1% DMSO (top), 0.15 µM panobinostat (center), or 12 µM prostratin (bottom). Numbers to the right of the blue bar indicate percent GFP-positive (i.e., HIV-expressing) cells. (B) Effects of control LRAs on HIV-1 provirus expression in J-Lat cells. (C) Representative effects of compounds on apoptosis in Jurkat cells, as measured by annexin V staining. Numbers to the right of the blue bar indicate percent APC-positive (i.e., apoptotic) cells. (D) Effects of control LRAs on apoptosis in Jurkat cells, expressed as fold-increase in apoptosis relative to control cultures treated with 0.1% DMSO (dotted line).
Figure 1. HIV-1 latency reversal and cellular toxicity assessments of control LRAs. (A) Representative GFP expression in J-Lat cells treated with 0.1% DMSO (top), 0.15 µM panobinostat (center), or 12 µM prostratin (bottom). Numbers to the right of the blue bar indicate percent GFP-positive (i.e., HIV-expressing) cells. (B) Effects of control LRAs on HIV-1 provirus expression in J-Lat cells. (C) Representative effects of compounds on apoptosis in Jurkat cells, as measured by annexin V staining. Numbers to the right of the blue bar indicate percent APC-positive (i.e., apoptotic) cells. (D) Effects of control LRAs on apoptosis in Jurkat cells, expressed as fold-increase in apoptosis relative to control cultures treated with 0.1% DMSO (dotted line).
Viruses 10 00348 g001
Figure 2. Discovery of new LRAs from marine natural products. (A) HIV latency reversal in J-Lat 9.2 cells, as assessed by measuring GFP reporter expression in the presence of 250 compounds from marine natural products at 2.5 µg/mL. Colored bars denote 4 of 250 compounds (1.6%) that induced GFP expression in at least 4% of cells (dotted line). (B) Structures of 4 LRAs identified from screening of 257 pure natural products at 2.5 µg/mL. Colors correspond to bars in panel A. (C) Representative GFP expression in J-Lat 9.2 cells treated with 3.8 µM psammaplin A (top) or 3.7 µM aplysiatoxin (bottom). Numbers to the right of the blue bar indicate percent GFP-positive (i.e., HIV-expressing) cells. (D) Effects of LRAs on HIV-1 provirus expression in J-Lat cells. (E) Representative effects of compounds on apoptosis in Jurkat cells, as measured by annexin V staining. Numbers to the right of the blue bar indicate percent APC-positive (i.e., apoptotic) cells. (F) Effects on apoptosis in Jurkat cells, expressed as fold-increase in apoptosis relative to control cultures treated with 0.1% DMSO (dotted line). Data presented in panels C and E originate from the same experiments shown in Figure 1A,C, respectively.
Figure 2. Discovery of new LRAs from marine natural products. (A) HIV latency reversal in J-Lat 9.2 cells, as assessed by measuring GFP reporter expression in the presence of 250 compounds from marine natural products at 2.5 µg/mL. Colored bars denote 4 of 250 compounds (1.6%) that induced GFP expression in at least 4% of cells (dotted line). (B) Structures of 4 LRAs identified from screening of 257 pure natural products at 2.5 µg/mL. Colors correspond to bars in panel A. (C) Representative GFP expression in J-Lat 9.2 cells treated with 3.8 µM psammaplin A (top) or 3.7 µM aplysiatoxin (bottom). Numbers to the right of the blue bar indicate percent GFP-positive (i.e., HIV-expressing) cells. (D) Effects of LRAs on HIV-1 provirus expression in J-Lat cells. (E) Representative effects of compounds on apoptosis in Jurkat cells, as measured by annexin V staining. Numbers to the right of the blue bar indicate percent APC-positive (i.e., apoptotic) cells. (F) Effects on apoptosis in Jurkat cells, expressed as fold-increase in apoptosis relative to control cultures treated with 0.1% DMSO (dotted line). Data presented in panels C and E originate from the same experiments shown in Figure 1A,C, respectively.
Viruses 10 00348 g002aViruses 10 00348 g002b
Figure 3. Natural products reverse latency in multiple cell line models. (A,B) Effects of LRAs on GFP expression in J-Lat 8.4 (A) and 10.6 cells (B). (C) Effects of LRAs on intracellular viral p24Gag expression in J-Lat 10.6 cells. Data are presented as fold-change in supernatant p24Gag relative to cells treated with 0.1% DMSO vehicle control (dotted line).
Figure 3. Natural products reverse latency in multiple cell line models. (A,B) Effects of LRAs on GFP expression in J-Lat 8.4 (A) and 10.6 cells (B). (C) Effects of LRAs on intracellular viral p24Gag expression in J-Lat 10.6 cells. Data are presented as fold-change in supernatant p24Gag relative to cells treated with 0.1% DMSO vehicle control (dotted line).
Viruses 10 00348 g003
Figure 4. Effects of LRAs on peripheral blood mononuclear cells (PBMCs) from three HIV-infected donors. (A) p24Gag viral protein levels in PBMC culture supernatants following 24 h treatment with control compounds or candidate LRAs for 24 h. Data are presented as fold-change in supernatant p24Gag relative to PBMCs treated with 0.1% DMSO vehicle control (dotted line). (B) Viability of PBMCs after 24 h treatment with LRAs, as measured by Viacount cell-permeable dye. Data are presented as viability relative to PBMCs treated with 0.1% DMSO vehicle control (dotted line). In both panels, shapes represent PBMCs from individual donors.
Figure 4. Effects of LRAs on peripheral blood mononuclear cells (PBMCs) from three HIV-infected donors. (A) p24Gag viral protein levels in PBMC culture supernatants following 24 h treatment with control compounds or candidate LRAs for 24 h. Data are presented as fold-change in supernatant p24Gag relative to PBMCs treated with 0.1% DMSO vehicle control (dotted line). (B) Viability of PBMCs after 24 h treatment with LRAs, as measured by Viacount cell-permeable dye. Data are presented as viability relative to PBMCs treated with 0.1% DMSO vehicle control (dotted line). In both panels, shapes represent PBMCs from individual donors.
Viruses 10 00348 g004
Figure 5. Effects of LRAs and LRA combinations on HIV-1 provirus expression in J-Lat cells. * p < 0.05 using the Bliss independence model [10,11].
Figure 5. Effects of LRAs and LRA combinations on HIV-1 provirus expression in J-Lat cells. * p < 0.05 using the Bliss independence model [10,11].
Viruses 10 00348 g005
Table 1. 50% effective concentrations (EC50s) of latency-reversing agents (LRAs). EC50s were calculated in J-Lat 9.2, 8.4, and 10.6 cells based on the percent of GFP expression relative to controls treated with 12, 38, or 3.8 µM prostratin, respectively, using the approach of Hashemi et al. [28]. n.d., not determined.
Table 1. 50% effective concentrations (EC50s) of latency-reversing agents (LRAs). EC50s were calculated in J-Lat 9.2, 8.4, and 10.6 cells based on the percent of GFP expression relative to controls treated with 12, 38, or 3.8 µM prostratin, respectively, using the approach of Hashemi et al. [28]. n.d., not determined.
LRAEC50 (µM; mean ± s.e.m.)Mechanism of ActionRef.
J-Lat 9.2J-Lat 8.4J-Lat 10.6
Panobinostat0.10 ± 0.020.073 ± 0.0100.041 ± 0.003HDACi[5,6]
Prostratin7.1 ± 2.810 ± 11.8 ± 0.4PKC activator[5,6]
Psammaplin A1.9 ± 0.31.5 ± 0.11.5 ± 0.1HDACi[29,30,31]
Aplysiatoxin0.045 ± 0.0210.011 ± 0.0030.0033 ± 0.0012PKC activator[32,33,34]
Debromoaplysiatoxin0.92 ± 0.140.52 ± 0.020.081 ± 0.029PKC activator[32,34]
Alotaketal C1.3 ± 0.2n.d.n.d.PKC activator[14]

Share and Cite

MDPI and ACS Style

Richard, K.; Williams, D.E.; De Silva, E.D.; Brockman, M.A.; Brumme, Z.L.; Andersen, R.J.; Tietjen, I. Identification of Novel HIV-1 Latency-Reversing Agents from a Library of Marine Natural Products. Viruses 2018, 10, 348. https://0-doi-org.brum.beds.ac.uk/10.3390/v10070348

AMA Style

Richard K, Williams DE, De Silva ED, Brockman MA, Brumme ZL, Andersen RJ, Tietjen I. Identification of Novel HIV-1 Latency-Reversing Agents from a Library of Marine Natural Products. Viruses. 2018; 10(7):348. https://0-doi-org.brum.beds.ac.uk/10.3390/v10070348

Chicago/Turabian Style

Richard, Khumoekae, David E. Williams, E. Dilip De Silva, Mark A. Brockman, Zabrina L. Brumme, Raymond J. Andersen, and Ian Tietjen. 2018. "Identification of Novel HIV-1 Latency-Reversing Agents from a Library of Marine Natural Products" Viruses 10, no. 7: 348. https://0-doi-org.brum.beds.ac.uk/10.3390/v10070348

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop