text:
HashiCast Episode 8 - Daniel Bryant, Datawire
Sep 25, 2018
This episode of HashiCast features Daniel Bryant, Product Architect at Datawire. Join us as we talk to Daniel about his career and his experience in the technology field over the years. We also dive into the world of Service Meshes and how they help with your application architecture. We also talk about API Gateways, Kubernetes and most importantly about defeasible logics.
- Daniel BryantProduct Architect, Datawire
This episode of HashiCast features Daniel Bryant, Product Architect at Datawire.
Join us as we talk to Daniel about his career and his experience in the technology field over the years. We also dive into the world of Service Meshes and how they help with your application architecture. We also talk about API Gateways, Kubernetes and most importantly about defeasible logics.
- Datawire's Ambassador
- Daniel's blog post on Distributed Tracing with Java "MicroDonuts", Kubernetes and the Ambassador API Gateway
- Daniel's Medium profile
- Nic JacksonDeveloper Advocate, HashiCorp
- Anubhav MishraTechnical Advisor to the CTO, HashiCorp
Welcome to HashiCast, the self-proclaimed number one podcast about the world of DevOps practices, tools, and practitioners.
Nic Jackson: Welcome to another episode of HashiCast. It's a real pleasure to introduce today's guest because we've got Daniel Bryant from Datawire. Daniel's had an incredibly interesting career, and currently he's working as a product architect for Datawire, and doing some really great things with Ambassador, which is their API gateway, and which integrates with Envoy. He's written Continuous Delivery with Java for O'Reilly Books. He's one of the InfoQ editors. He's a renowned conference speaker, and one of my all-time favorite people. Welcome, Daniel. I could have filled the entire show with just an introduction of your achievement, but you can probably do a better job for me. Why don't you introduce yourself?
Daniel Bryant: Thanks, Nick. My mom would be proud; if she actually hears this podcast, she would love this intro. Thanks. Obviously, you and I have worked together in the past, and currently we see each other on the community trail, on the conference trail as well.
A little bit of background about myself: I've been a software developer in some form or another for about 15 years now. Worked for the UK government my first gig, and then I went to work with a bunch of startups. I liked the fast feedback loop, if I'm honest, working with the startups. I worked for the UK government, and as fun as it was, 18 months I was there, and didn't actually deploy anything to production. That kind of sucked.
And then went to a startup and literally pushed code, and then the user was complaining to me the next day. Literally they were complaining. I loved it. I love the fast feedback. I loved just learning everything I could. It was like, "Hey, we don't have someone to run a database. Do you want to learn how to do database stuff?" And I was like, "Sure thing." So I did a bunch of that for a while, did a couple of CTO gigs at various startups, SpectoLabs. And there Nic and I again bumped into each other on the circuit there when I was doing some talks around that.
And then I went back to consulting, did a bit of consulting with a company called OpenCredo. Shout out to the OpenCredo folks in London. They're HashiCorp partners, actually, and you guys probably bump into them a lot, but they're real good people. Still keep in contact with Nicki there. I was just chatting to her last week, actually.
And after I left OpenCredo, I fancied a new challenge. I fancied getting back more into the product as opposed to the services. I loved doing the consulting and stuff. Loved solving problems, but I think my credo probably banks between services and products.
At the moment, working with the awesome folks at Datawire. They're based in Boston, although I'm based in the UK. And we are trying to create some cool, interesting, and ultimately useful tools for Kubernetes, because we're thinking, "There's awesome tech now with Kubernetes, and cloud, and stuff, but we haven't fully, as an industry, caught up with our practices on how to deploy and deliver code and value.
I'm loving the challenge. I'll be honest: It's a big challenge. I'm sure everyone working at HashiCorp, and I'm sure everyone listening, knows these challenges. We've got amazing stuff out there, but I'm really enjoying the next stage of my career, trying to help people who are developing code, developing infra-tools and stuff, get the best out of what they're trying to do.
Nic: Now, Daniel, you're a little bit modest, because—I didn't realize this until I saw a tweet—but you're actually a doctor. You are Dr. Daniel Bryant.
Daniel: Yeah. I only use that when I'm flying. Someone told me put that in when you're buying a flight, you might get an upgrade. The only time I actually use the doctor thing is when I'm buying flight tickets. But yeah, Nick, I did a PhD in 2004, 2005, in defeasible logics. I'm sure this is very much a niche audience, and to be honest …
Anubhav Mishra: What is that?
Daniel: Basically, classic logics is A implies B, that kind of stuff. With classic logics, it's monotonically increasing. You can't almost unlearn. You draw a conclusion, you draw an inference, you can't contradict that easily. Whereas we all know in the real world, you learn new stuff that invalidates old stuff, and you also make tentative guesses. The classic joke with the defeasible logic crowd—and trust me, this is the best the jokes get—but the classic joke is, "All birds fly, but what happens when you get a penguin?" Penguin is a bird; penguins don't fly. And that is a real trivial and contrived example, but there's a whole bunch of research going on in developing algorithms for modeling, in particular the legal and the medical side.
I did a lot of work with oncology doctors, a lot of work with cancer patients, genuinely fascinating and equally respectful, and scary at some times. Whole bunch of work with lawyers, and trying to capture the way they reasoned about things in computers. The classic expert systems are very much if-then, rule-based, whereas the real world, as we're seeing with the rise of ML and AI and the stuff now with deep learning—and deep learning is one of ways of looking at these things. Back 10 years ago, we were looking more at the argumentation theory, it was called, the feasible logics and nonmonotonic logics.
And to be honest, it was super interesting, but it was very niche. And one of the big things that caused me to move on from that was it was very theoretical. The people I was working with were super into proving one thing, disproving another, and I was just like delivering value. It's a cool strand in my career. I just want to actually do real stuff that helps people rather than trying sit around a dinner table, and try and solve the world's problems just by talking. I actually wanted to do stuff. As much as I love my academic peers—I'm not trying to knock them—it just wasn't for me. I thoroughly enjoyed, learned a bunch of stuff, but I wanted to actually get out there. And when I started doing consultancy part time, that totally led me down the coding route there.
Nic: That's fascinating, and unfortunate, because I'm not going to be able to ask you about this pain I've got in my leg right now.
Daniel: I get that joke, Nick, yeah, because it is rash.
Nic: We should probably get back to the tech. I'm interested in your role at Datawire because obviously you were saying you're moving back into the product architecture side of things. But you're an incredible software architect, and a systems architect. How do those skills translate into product architecture and into creating great, great product?
Daniel: Thanks for the compliment, Nick. Obviously, you and I cut our teeth on some of the microservice stuff together, didn't we, back at notonthehighstreet.com. I think one of the key things, and it's another thread in my career which I've gradually learned, is empathy, a bit of a catch-all. But being able to put yourself in someone else's place. In life, I think this is just super valuable.
But in particular with the kind of stuff I'm doing now, working a lot with Richard and Rafi, who run Datawire, I think some of the value I add is being able to go, "Hang on. That's not how I've done it in the field." Although it's kind of anecdotal, got to be careful with that. But I'm super lucky with the InfoQ stuff, super lucky with the conference stuff I do. I get to talk to people from Netflix, people like you guys. I get to hang out with super-interesting people, and I learn a bunch of stuff. And I combine that with some of the things I've done right, some of the things I've done wrong, some of the teams I've led, some of the mistakes and good things we've done, match it all up together. And then it gives me a little bit of an edge in terms of thinking about, "Will the product actually meet the needs?"
The whole Datawire team is amazing, hats off to them. But Richard, for example, is super good at business. Richard looks at the business problems. He's super techy, but he focuses on the business. Rafi and the technical team, the engineering team, look at engineering top-quality products. I'm in the middle going, "Let's think about the user." All of us in Datawire bring different things, and I'm only mentioning three names there.
But I think that's value I add is being able to go, "Hey, what you often think in your little bubble sometimes, as good as it is, we think, 'Oh, this will be amazing.' And you roll it out to users, they go, 'What were you thinking?'" Because in the real world when the rubber meets the road, as they say, it's a different experience.
That's probably the thing I've noticed the most about having the experience of actually doing these things, coding these things—and making mistakes, to be honest, learning along the way.
Nic: That somewhat segues into my question around service meshes, because that's obviously a big part of what you're doing with Ambassador, and I'm going to say I was incredibly skeptical when I first heard of service meshes. And I think I had this conversation with you before.
Daniel: Yeah, you did.
Nic: Because I looked at it from a microservices practitioner's problem, and I'm like, "Well, I don't actually personally believe that implementing the classic, historic circuit breaker, load balancer retrier, back-off type pattern was really a problem when you're building out systems." And actually, I quite like the control it gave me. This was one of my main reasons why I'm like, "Well, hey, why do you need a service mesh?" From that empathy perspective, I think one of the things that I came to terms with is that everybody is different, and everybody has different needs, and actually service meshes have got a very good place in the market to lift a lot of that load away from developers. I actually now believe they're essential for most systems. And I'll completely retract any bad words I've said about them in the past. I'm happy to admit I was wrong.
But what are your views on the service mesh? Because you've spoken about it as well. I watched a video you did for O'Reilly, I think, a webinar on service meshes. What do you think the benefits are? Why do you think they maybe fall short? Are they the silver bullet that everybody is looking for?
Daniel: A whole bunch of interesting questions there, Nick. One of the things, just to unpack some of that, is I definitely think many different people have different views on what platform, what application, what's useful, these kinds of things. My experience working in startups is you're often happy to put everything in the code, in the application, and that works for you because you're going really fast. Whereas when you're an enterprise, you have separation of concerns, both from a skill point of view, but sometimes from a regulation point of view. You want to have that "this is platform, this is cons, this is application" type vibe. I find it useful. OpenCredo is really good in this respect.
But going into some big companies, I could never understand why people used CSB sometimes, but then in some big organizations, financial organizations, it was essential. The sheer scale of the people working on it, you had to integrate at some point, and you had to draw the line, and say, "This CSB is going to be our gold standard of how we transport messages, how we route messages." All these kinds of things. It really gave me some empathy for understanding why all these different tools exist.
And if you look at it from both angles that we saw when we were at Not on the High Street. It was primarily a Ruby shop at the time, doing a lot of Ruby on Rails. We were bringing in some Java into the stack as well. I remember distinctly we loved the ideas of Hystrix. And we plugged in the Netflix OSS Hystrix, and the Java stack—shout out to Will, awesome developer at OpenCredo and Not on the High Street—and he hooked it all up: "Oh, that's amazing, full backs, circuit breaking, love it."
Then we wanted to do the same with the Ruby side, and we found a gem, we found a Hystrix gem, but it was proper janky. By their own admission, they'd literally written it in their bedroom over a weekend, chucked the code up, and they were like, "By the way, caveat emptor." And we took it on, and it was real hard to get it up to the standard of the Java version of the Hystrix. And we also had the same issue with service discovery. That was when I first bumped into Datawire, actually, with Baker Street, but we also looked at Airbnb SmartStack, and GeneXus, and a whole bunch of tools.
I suddenly realized that there's a whole bunch of communication cross-cutting concerns, which when you started implementing them in libraries, unless you were a shop like Netflix that really went all in on the JVM or had a massive engineering team that allowed you to create JVM slide casts for other language stacks to use, it was actually really hard to get this stuff right. You had to re-implement the same things in many different languages and libraries. We were looking at bringing in JavaScript and Go and Swift. And the thought of maintaining the libraries filled me with dread a little bit, in terms of, "If we change the service discovery mechanism in the Java world, we're probably going to have to update the Go version and so forth."
That's why I think the service meshes feel like the missing piece from the Kubernetes and orchestrator puzzle. A bunch work on Mesos, a bunch of work with Kubernetes ECS—fantastic tools for deploying onto ephemeral hardware, stuff that dies all the time, and that kind of stuff. Kubernetes will reschedule your pods and Nomad—same kind of deal. All these tools are fantastic at that. But they left the communication bit as an exercise for the reader. You walked up to Kubernetes, and you're like, "Oh, this deployment stuff is amazing." Then you start trying to do canary rollouts—bit hard sometimes. You start trying to do service discovery. The DNS is good enough, but doesn't always get what you want. You want to start putting security policies in; you're out of luck in some ways.
I looked at Toygaroo, they were doing a bunch of stuff, and Weibo obviously doing a bunch of stuff, Calico, and things like that, and Weave Net. That was interesting, but again it was pretty low-level. I think the service mesh has come in almost above that proper Layer 3 networking thing, and are complementing Layer 4 upwards, like Layer 4, 5, 6, 7, and really centralizing a bunch of cross-cutting concerns in a platform level. It's language-agnostic, if that makes sense. I've used it in a bunch of gigs. Once you get your head around it, it's been very useful.
Nic: I mean, as I say, I'm a convert. But does it mean that you do lose control as a service developer? I'm not going to say an application developer, but a service developer. Because one of my key things I think is incredibly important is designing for failure, especially in a microservice environment, because they fail all of the time. They're robust, but independent components fail all of the time. And if you don't get that plumbing correct, then you can end up with a complete system failure, but if I'm putting all of this logic into the service mesh and the service mesh is handling my retries, can I control, for example, service-A-to-service-B has a different retry and back-off than service-D-to-service-B? Is that possible, or do you have to have a one-fits-all solution?
Daniel: It comes back to something you mentioned just a minute ago, Nic, which I didn't really address. And it's the silver-bullet thing, isn't it? I mean, we constantly search for silver bullets, and even Fred Brooks, Mythical Man Month, said, "There are no silver bullets," which I'm sure all of us that are listening can relate to as well. And I totally get what you're saying. I've struggled a bit with this, to be honest. There are some things that are application concerns, and the service mesh can almost take them away from you. It makes it easier to get on with some of these things. Rather than having nothing—like a service mesh clearly gives you often some service discovery, and it gives you some fault tolerance, and stuff that I'd rather have than not have.
But to your point exactly, particularly as an engineer, particularly if I'm doing a startup, I'm trying to go as fast as possible, I want everything under my control. And the service mesh can take some of that away. And the person I look to in this space is Christian Posta. I'm sure I'll mention Christian's name many times in this call because Christian is just awesome. I'm super lucky to have met him last year. We've had a bunch of interesting chats, and just recently he put out this awesome blog post. It was about applications safety and correctness, saying a service mesh can't do everything. And basically, Christian just nailed it. He's one of those people, a bit like a Matt Klein or even Mitchell and Armon, who can take something we're all thinking and put it in a really consumable package.
And when I read Christian's stuff, the way he's split it up, I think it was application integration and application networking, and he said, "The integration stuff is like an app-level concern, for you and I as developers, all of us here as developers, and the application networking is more of a platform thing." Now, the service mesh is awesome at the application networking and platform level, but you want to use other tools. I think Christian is a big fan of an old Java tool, Apache Camel, and I've used that as well. That one you can package up and do some funky application-level things as well.
I totally think it's a really good shout-out to think about these things. One thing I did get from a chat with Matt Klein at QCon New York is that he was using Envoy Lyft, and the team at Lyft were using Envoy to provide sensible defaults, like in concurrency and throughput and safety. And I bumped into some of that with Sentinel, for example, in some of the policy work I know you were doing at HashiCorp. And I think having these low-level sensible defaults, but then having the freedom and responsibility to override them by one level up, is where it's at. And it's not for every development team. You've got to be a bit aware and a bit responsible for these things. But I definitely think it's worth calling out, and then making the decision consciously: "Are we going to put some app-level integrations in? Are we going to rely 100% on the platform?"
Nic: I think maybe as well when I look at things, I'm maybe overrating the need for too much configuration. I think sometimes things work fine, and you can possibly over-evaluate things, but I think certainly the key thing is the fact that if you can just throw something with an annotation and automatically not have to worry about any of the things like load balancing and service discovery, that is an incredible movement for a team to be able to work fast, and to not have to concentrate on those things, and then concentrate on features, which deliver value.
Daniel: Yeah, yeah, I hear you, Nic. One thing I should chuck in there is it's always worth thinking about mechanical sympathy. As in, if you do get a bit divorced—because it's super powerful to be able to chuck in an annotation—but as a developer sometimes I lose touch with the platform. Something I constantly remind myself of is, every time a tool gives me a good abstraction or solves some problems for me, I still want to know about the problems so I can code appropriately.
If there's something my mentors have taught me over the years it's always look one level deeper. If you're operating as an architect, you've got to be able to code. If you're coding, you've got to understand the infrastructure to some degree. Maybe not write it, create it, but understand it. And I do warn that sometimes the service meshes might hide some of the responsibilities from developers. Not seeing it too much yet, but it's a potential, I think.
Nic: I think one of the things that I come across is around observability. And I think certainly, in some areas, there's maybe a mistaken belief that the service mesh can take away all of your observability woes and needs, where actually from experience and, well, from the project that we worked on, actually having observability deep down into the code is really where it helps to be able to do your diagnostics when you're either load testing, performance testing or if you're dealing with an outage. And certainly the level that you get with the mesh, the fact that you have this intercommunication, you can see number of calls, time of call, and stuff like that, it's incredibly valuable. But how does the service mesh take that further? How do you build up that big picture which is inclusive of the service-to-service communication, but also being able to include very granular service-level metrics, such as how long it took to execute it—a routine or a function or a loop which might be doing some sort of calculation, or just a database call? How do you concatenate all of that together and present it in a usable way?
Daniel: Great question, Nic. And it's a definitely a super-hard question, and to be honest, a super-hard answer. I think there's a bunch of people doing very interesting work in this space, but—a shameless plug—I did write a bunch about this stuff in my book. As in, not just plugging the book, but it really made me understand how much of a challenge it is. The chapter of the book just grew and grew and grew as I was writing about all this stuff.
Again, if you haven't got anything, adding a service mesh gives you a default view. Top-line metrics, are you getting 500s? What's the latency? What's the throughput, for example, of your service? That's good, but you also do need to instrument your applications. You're right. In the Java world, we have codahale.metrics, and Spring has got some, and I think Go has got equivalent versions of these. I often added semantically relevant data points, even sometimes business data points, to be honest. But even things like you mentioned. Super easy, actually, in the Java world with codahale.metrics. You put an annotation on your method, and it literally times the entry and the exit to the method. And you can then kick that up to Prometheus or something else as well.
Two people I'll shout out here that I've learned a lot from are Cindy Sridharan, who's done amazing work, @copyconstructs on Twitter and Medium, and Charity Majors, @mipsytipsy. Those two are really doing some amazing thought leadering, and Charity in particular with Honeycomb, a super interesting product she's coming up with. I was literally chatting with Richard at Datawire today about this. We're seeing there are two modes of operation. There is the observability piece, the monitoring piece, understanding on a global level what's going on at your systems, understanding service-to-service comms. Then there's an understanding of what's going on from a business perspective. Is the actual user getting value? Are they impeded in any way?
And then what I think Charity in particular talks a lot about is debugging. She calls it high-cardinality debugging, the ability to dial into a particular user. Maybe a high-value user, a HashiCorp premier customer, for example. You just say, "Oh, 1% of my traffic is getting bad results. Who cares?" But if that 1% is exclusively all of your top-priority customers, that's a real big problem.
Also, the LightStep team, Ben Sigelman, doing amazing work.
It's an area of innovation. People are chucking in some machine learning—I think that's genuinely good. You often see a bit of blockchain, a bit of machine learning in landing places, but I think in this place it's actually quite an interesting space to talk about insight. And as a developer, when something has gone wrong, I really would value someone or something to point me in the direction of where it's gone wrong. From what my friends at Google and so forth, and from conference presentations I've seen, Google with their SRE teams have really worked on this kind of thing, where the heuristics and maxims of the systems will guide you to where the problem is.
In the real world, the world outside of Google and Netflix and stuff, we struggle with this. People like Charity and Cindy are really helping us think about this, the observability, understandability, and debuggability. They're really three key things, I think, as an engineer.
Mishra: I remember looking into all of this when we were doing microservices at my previous job at Hootsuite. Our dream was to have tracing all the way from the browser, so from the browser you can actually see the traces for each API call, and you had multiple microservices doing different things. And we were a polyglot organization, so we were running Golang, Scala microservices, Python microservices. And it was like, "Okay, we need to instrument each of these microservices, then we need to pass these headers all the way through from the browser all the way down the stack." It was a big project. When we proposed this, we had some pros and cons written down, saying, "If we were to invest in this how long will it take? Is it even useful."
And I think we tried it for a subset of our microservice calls, and even then we saw so much benefit that comes out of this kind of system. But yeah, the organization has to be with this, and the organization should understand the pros and cons of this, and actually invest in this in order to make this a reality. It's not an easy buy-in to get from the engineering teams, like, "This is what we're going to contribute versus ..."
Daniel: It's something I struggle with. I was listening recently to Beyond the Phoenix Project, by Gene Kim and John Willis, an awesome audio book that builds on The Phoenix Project stuff. And they're a couple of people that crystallize thoughts I've been having for a long time. Say I was working in a chair manufacturing plant. You could see the trees coming in one end, the wood. And you could see the chairs popping out at the other end. You could see the broken chairs, and you can see you us taking a long time to put the back on the chair. But in software, you can't see any of that. And then trying to argue that this observability will be really valuable, trying to argue that to someone who is perhaps more of a business person than an IT person, there's no obvious analogy we can use here. We can't point to the trees. There are no trees in software.
This is something I as a consultant really struggled with initially. Often as a tech person, we don't look at the ROI or the risk. As a consultant, as a CTO, it's something I got a bit burned with, and I learned my lessons. When I was responsible for the profit and loss, I had exactly that conversation: "Do I really need this debugging stuff? Can I afford it?" But we need to think holistically about this stuff and come up with better analogies to help the not-so-IT-savvy people. Because we all can't be IT experts. We need to help them make these decisions, which is really hard.
Mishra: I'm glad now you work at Datawire. I feel like you're much more aligned with directly solving some of these problems, and I'm pretty happy that Datawire is actually doing all the products that I've used. I remember a year ago I used Ambassador, and we were evaluating API gateways in my previous company. We had this really old API gateway written in Java.
I'm not going to name the company's name that we were using, but then we were at the same time transitioning from Mesos to Kubernetes, and we were looking into Ingress controllers, and Envoy was new. And I think we were trying out Envoy. And then we found this open-source project out of nowhere. It was like, oh, Ambassador. This is an API gateway on top of Envoy. I was like, "Oh, this is exactly what we're looking for for our future API gateway or platform." Could you tell us a little bit more about what Ambassador is and what problems does it solve?
Daniel: Yeah, sure thing. That story is one I bump into nearly every day, Mishra, to be honest. Fundamentally, Ambassador builds on Envoy. It's written in Python, and Envoy at the core, but we've sprinkled some magic sauce over the top. But to make it easier to use, because Envoy is super powerful, but it's also quite challenging to configure sometimes. Lots of conflict files going in. Now they've got all their APIs a bit more locked in. It's a bit more straightforward, but one of the core things people find when they go to Kubernetes, either they're doing migration or they're doing some greenfield work. One of the first things you want to do is get stuff into your cluster, get traffic Ingress into your cluster.
And it's actually not as straightforward as you might think. I don't know if you've played around with the Ingress options in Kubernetes.
Mishra: I have.
Daniel: And they were a great first start, but some of the discussions around them have stalled, and now Ingress is a little bit wanting compared with, say, the load balancer on the services. We saw a bunch. The default people often go to is using Nginx because everyone knows and loves NGINX, as do I. Used it many times. We used it at Not on the High Street with Nick. But NGINX isn't very cloud-native, at least not at the moment. But all of the things I'm about to mention, depending on when this podcast is published, it probably will all be changed. NGINX will be super Kubernetes-native and stuff. But at the moment, it's not. And as much as I love NGINX, you've got to learn the syntax of the conflict files. And if you're a developer, you're having to learn Kubernetes already. You're having to learn YAML, perhaps, and learn Kubernetes. To learn something else is just more and more stuff.
We deliberately said—I say, "We" but it was before I actually joined Datawire—"How can we make Envoy easy for people that are comfortable with Kubernetes?" And ultimately now it's gone to using annotations. You've got your standard service. You want to expose it via a route, an endpoint, whatever. And you literally write an annotation in Kubernetes, and you can do the standard stuff you'd see on any gateway really. You map a route to a service, wild cards, all these kind of good things. It fundamentally is there to make people's lives easier.
We're looking at things like CRDs. There's been a bit of work by the Admiral team actually doing a CRD. It was making it even more Kubernetes-native, how you define these routes and things. And we've also got a bunch of stuff; because we're building on Envoy, you can actually access the Envoy effectively, but we often put a nicer abstraction on top. We've got integrations that do, say, rate limiting. We use a GRPC interface that Envoy expects, and you can create a service in any language, as long as it builds on the GRPC interface, and you can do things like rate limiting. I've got a couple examples in my Medium account of where I built a Java rate-limiting service that you can do customization or various properties based on the request and things.
Same kind of deal with auth. The Kubernetes Ingress is a bit funny with auth. Istio have totally recognized this problem. Istio have got gateways now. They're doing some fantastic work around the gateways abstraction in net service mesh, but even they don't support authentication yet. And there's a GRPC interface again with Envoy that we expose in Ambassador, so you can write your own auth plugin. We've got a pro version where we actually will code the integration for you, and you can integrate with your Active Directory or whatever you want to do.
We're trying to, without sounding cliché in some ways, bring Envoy to the masses. Because it's totally cliché. Matt Klein has already brought Envoy to the masses. I wouldn't dare get in the same ring with Matt Klein. He'd do this at another level above, but we wanted to make it easier for people to get all the awesome benefits from what they're doing, and use it on the edge, as well as on all their east-west traffic, all their service mesh traffic. So we were like, "The first problem is often Ingress. Let's help people get that one done."
Mishra: I think that sounds really interesting, because Mitchell and I were just talking in our Kubernetes internal group about CRDs and how you can use RBAC on top of CRDs to do some really cool stuff. And I think one of the major reasons why people vent from using, let's say, config maps or something else to store all that information using CRDs is RBAC. And that's super interesting to me that slowly we have come to a consensus. I think Ingress is a great example that you mention. I felt that on a control plane it was trying to do a lot of things for a lot of different proxies.
Daniel: Yeah, there you go, perfect.
Mishra: It became really difficult to write these configurations. Like, "Oh, this has to work across every data plane." And it becomes so difficult. And I think Ingress is still useful, and I'm sure there's a grand plan for what Ingress will be, but I'm glad there are products like Ambassador and it's all open source, and you can use it. I think we used it for a substantial amount of time.
Daniel: Yeah, perfectly said, Mishra. Definitely I think one of the challenges with a project like Kubernetes is you want to welcome everyone with their different solutions, but if you're not careful you fall back to the lowest common denominator. I've definitely seen that with Ingress. Everyone was like, "We want it super configured, but we want it generic." That's hard. That's a genuinely hard problem to solve. The Kubernetes community is fantastic. There probably isn't a stronger community, but you have to draw the line somewhere, I guess. And that's what we've tried to do with Ambassador. We can because we're outside, working very closely with Kubernetes, but outside. We can draw the line and say, "Hey, here's our opinionated version." And it's open source as you rightly said. And getambassador.io is the website to hop along to. People can read more about it, and download it, and just try it out. If it doesn't work for you, totally cool, but we find it's useful for a lot of people.
Mishra: Yeah, we'll drop the link down in the description for the listeners, and also drop some of your blog posts from Medium. I think you mentioned two of them that talk about Java and things like that.
Daniel: Yeah, thank you.
Mishra: That would be super useful. I always love reading them.
Daniel: Appreciate it.
Mishra: One other question I had around Ambassador was the integration or even the interplay between service meshes and Ambassador. How does that work? Do you have to have a service mesh in order to use Ambassador? Or do you just need Envoy and then maybe a control plane or a controller that Ambassador provides?
Daniel: Good question. I was in Dublin last week doing a talk, and I bumped into this question a couple times after the meetup. Ambassador is standalone. So if you're just looking to get traffic into your network, Ambassador will sort you out completely. Often people start, say, with a migration with a monolith, like we did on Not on the High Street. You chip a bit of the monolith off. You create a couple of services, and your service call stack is often quite shallow. Ambassador is perfect there because you can literally, as you would with any other Ingress gateway, route traffic accordingly and to those shallow services and your monolith.
Then, when you grow in complexity, often you might want to look at a service mesh. My general rule of thumb is if you've got less than say 10 or 20 services, a service mesh might add overhead that you don't quite get your return on it. But that's an anecdotal thing—still working around this kind of stuff. But, if you check up our Docs app, a bunch of people have integrated Istio with Ambassador, for example. The control planes are still separate, so we have the annotations on the Kubernetes services doing all the routes for the Ingress, and we have an Ambassador admin service or pod and pods deployments that do all the magic there at the control plane. You still need to run Istio with all its mixer and pilots and everything, so you do have two control planes. And you need the Istio routing, which is obviously done differently. They have a separate config file.
But we've had a bunch of people do mutual TLS, for example. End-to-end, Ambassador does the TLS termination, then it will pass onto Istio, and Istio does mutual TLS internally. I know we're looking at doing stuff with Consul Connect, but as soon as I bumped into Consul Connect, I reached straight out to Nic. And I was like, "This is super interesting." The Envoy stuff for us is really interesting because we're doing a lot of stuff with Envoy, so we're waiting to see what's going on around the Envoy work there.
But regardless of what mesh you choose, I think there's going to be meshes for almost all different use cases. I think the mutual TLS stuff that Consul Connect is focusing on is a primary use case for a lot of enterprises. The SMEs I'm seeing their primary use case with SO is mesh's observability, and there are a couple other use cases. You might choose your mesh depending on what you value the most, but with Ambassador, we would look to integrate with all of them. It's going to be interesting. The biggest challenge I think around this stuff is the control planes. As a developer, I'm having to learn lots of stuff all the time, having to learn how Istio works, Ambassador works, Kubernetes.
I think as an industry, we need to come up with a bit of a better approach to some of that stuff. But we're totally up for it. We like to say we manage the north-south traffic, the traffic coming in and out of your cluster. But service meshes are where it's at for the east-west or cross-data center traffic.
Mishra: That makes complete sense. I think it's good you mention that our goal is to write software applications that we write for our business organizations. It's not to learn eight different technologies and trying to make them interplay. I think that's also important, but I feel like the goal in the end was to actually write good code, and hopefully that code brings in a certain amount of revenue.
Daniel: I saw Gareth Rushgrove today tweeting; he's doing the reviews for QCon, I think it was. And he was like, "If you're going to pitch me a service mesh solution, tell me how this adds value." It was a really pithy tweet. Exactly what you said, Mishra. The primary goal of all of us: We want to do cool tech and we want to work with cool people, but bottom line is we are delivering some form of value to users. And as an engineer I forget that. I'm sure we all do. But a reminder every so often that you're actually here to help the users focuses your attention on the actual business code.
Mishra: Talking about writing and testing software out, I really struggled with the initial phase of my learning when I got into Kubernetes. Minikube was still a new project that you could use to get a local Minikube cluster running. We had these really big Kubernetes clusters, both in a development environment and a staging environment and a production environment. How do I make things like integration tests work locally, unit testing, and things like that? I think it was really difficult for us to figure out a consistent way to do it, and we came up with our own way. And then I remember this project—again, Datawire was one of the people behind it, no surprise—called Telepresense. And I think initially when it came out, I remember it was on Hacker News. I think it was front page or something.
Daniel: That's always good.
Mishra: It's always nice looking through all the comments. It solved our problem at that time, and we didn't have to do eight different things to make, let's say, hijacking DNS, and that stuff that we're already doing in the background to hack this, make this integration testing work. This tool actually did it in a very consistent way. My question was more around how did you find this problem and how did you go about solving it in terms of using, let's say, Telepresence? But what do you see in the wild in terms of the Kubernetes community or, let's say, the cloud-native community? Where are these problems, and how do you go about finding them? How do you go about solving them?
Daniel: If I remember my Datawire history correctly, it was Itamar. (Itamar has actually moved on to do some other stuff now. He followed his dreams looking at some socially responsible companies, doing some software development there.) But he was basically brainstorming. One of the problems he constantly bumped into as a developer was that he was doing stuff like working on bigger projects in Kubernetes or whatever, and he couldn't spin them up on his local machine. He was like, "How do I locally develop a service but interact with all these services that I can only run on a remote cluster?" I think that's it if I got my history right, I think that was the general vibe. And it basically went from there. And Telepresence is actually a CNCF project now, so it's stewarded by Datawire, but it's CNCF, the Cloud Native Computing Foundation, a hosted project, which is awesome. Got a lot of great benefits from them in terms of building the community.
But fundamentally, Telepresence allows you to two-way proxy to a remote cluster. And the way I crudely look at it, because I've used it a bunch, is I can basically put my laptop in a remote cluster. The joke is it's cheaper than visiting a data center. I don't have to visit the GKE. I'm a big fan of using GKE and ephemeral nodes. Doesn't cost that much. I spin up all my config or whatever, and often because I do a lot of Java stuff, as much as I love Java, it's often quite thirsty on the RAM. So even though I've got a 16GB Mac, I can't run all my services on my Mac.
So I spin up everything in GKE on ephemeral nodes, and then I Telepresence into the cluster. And there's a nice command line flag you can do: swap deployment. And I can literally swap the deployment that's running remotely to my local machine. And I can then debug. I often use IntelliJ, for example, in the Java world. I can poke some traffic in through Ambassador, say, like in the real world, the actual remote cluster, poke some traffic in. And I can set some debug points on my swapped deployment. Basically the Telepresense will see there's some traffic due for my service. It will route the traffic to my local machine. I do my debugging and as long as the timeouts are set rightly, I can then send the traffic back into the cluster. And it will do its thing.
There's a whole bunch of interesting use cases there. I wouldn't recommend it for debugging live traffic, per se, although we are looking at that kind of thing, using funky things with headers you can shadow, and canary, and reroute traffic. But my primary use case is when I'm, say, messing around with a big local development—local remote development environment—or a staging cluster. And so it goes to the conversation we had earlier on is when stuff is going wrong when you really need to get in there and actually look under the hood, and figure out what is broken, Telepresence is amazing. You can run all your tools locally as if you were in the cluster. I'm often digging in DNS lookup and checking that all the DNS is done right. And then I'm literally debugging using all the tools of the Java dev that I love locally, but I'm primarily testing against their remote invocation, which has been routed via my laptop.
At Datawire and other companies I've worked at, but Datawire are really good at this, we've figured out the common problems. By working with customers, going and seeing their problems, you soon identify the pain points. When you've seen a few different customers, you spot the commonality. Rafi at Datawire has done a bunch of consulting gigs, and he was like, "We're seeing this at every gig. We should probably create a tool to outcome the issue." I'm pretty sure it's the same with HashiCorp.
I remember when I first used Vagrant, for example, I was like, "This is exactly what I wanted." Mitchell pretty much said, "I was scratching my own itch. I had this problem. I created a tool." We've inadvertently followed in his footsteps in that one.
Mishra: Thanks, Daniel. This is actually a very interesting discussion. I love asking this to all our guests, and I think one of our last questions is, "Where do you see the tech industry going?" This is a super-subjective question. We get a variety of answers.
Nic: What Mishra really wants to know is, Are robots going to take our jobs?
Mishra: "Will I still get paid?" is what I'm asking.
Daniel: It's a multi-level question, Mishra, to be fair. I work with InfoQ, and our emerging trends are things like blockchain, quantum, and augmented reality. They're super interesting, but for most people listening, and probably ourselves, in the actual development space, I think serverless is super interesting. My view is that serverless is going to coexist with containers for a while. Some people are saying, "Serverless is the only way to do things." And they're very clever people, and I respect them a lot, and I get where they're coming from, but I also can see the benefits of where we're at in terms of containers and stuff.
I think Kubernetes being the default fabric, if you like, the default platform fabric, things like Knative being built on top, and a bunch of—I'm sure, HashiCorp are doing lots of super-secret projects, which we'll get to see in a few months' time—I think it's a very interesting space around the serverless stuff.
And directly related to that, something we've already touched on today is the control plane. As developers, we need to be able to understand what's going on, and then interact appropriately. At Datawire, the infrastructure we've got access to is incredible: cloud, containers, Terraform, it's stuff I couldn't have dreamed of a few years ago when I was developing code.
But we're almost struggling to keep up with this in a development principle point of view. How do I actually use all this stuff? How do I test it? How do I deploy it? These kinds of things. Matt Klein has talked about this kind of stuff a lot. I think building effective control planes for observability, and understandability, and how we deploy and manage containered stuff, and serverless stuff is where it's at.
And that's why I joined Datawire. They were making bets on that being a very interesting area of development. That's where we're really looking to add a lot of value-on, on how you actually use all these things that deliver cool stuff.
Mishra: I don't want you to go without giving you an opportunity to plug your new book, which is coming out probably the same time this podcast comes out, I think. Could you tell us a little bit more about your new book?
Daniel: Yeah. The book will be out end of October. I like to joke it's a perfect holiday gift. Thanksgiving, Christmas, whatever you celebrate, I think it's a perfect gift. But, yeah, thanks for the opportunity, Mishra, and Nic just shouted the book.
It's with a colleague of mine, a friend of mine, Abraham Marín-Pérez. He joined me midway, did a fantastic job of really helping with so many things. Both of us have written the book. It aims to be a one-stop shop for a Java developer, be they fresh out of college or a seasoned Java developer looking to embrace the modern ways of working.
Something both Abraham and I have bumped into is all we've talked about today: working with cloud, working with containers, working with—maybe not service meshes. We haven't covered that too much. But definitely cloud and containers, and the demands of users these days, the fast feedback loops required. There's a whole bunch of stuff that we've got some best practices or some good practices around. Just in my consulting experience, they're not always well shared out. The future is here; it's just not evenly distributed, as the quote goes.
What we've tried to do in the book is capture our learnings. , and actually shouted you out, Nic, definitely. We learned a bunch of stuff together there. Abraham has done a bunch of work with Equal Experts—same kind of thing. We've tried to package it all up and provide a guideline for understanding all the stages of continuous delivery, how you develop things locally—Telepresence again gets a shout-out and these kinds of things—how you do testing. Even simple things like how you manage your Git workflow. I've definitely bumped into people who are fresh out of college and haven't done much version control, for example. We've tried to put a starter chapter there for people that are new to all these professional ways of developing software. Stuff that we all had to learn the hard way as we got into these things. There's a whole bunch of GitHub examples. I'm still writing some of those. We are using Packer, Vagrant, Terraform (you'll be pleased to hear) for our demos, bunch of Kubernetes stuff, bunch of ECS, EKS, I think as well, and some other things.
We've tried to make the book concrete, but with plenty of conceptual stuff, high-level stuff, and then the examples are actually going to be code-driven. But that is a work in progress. I'm literally coding this weekend, and probably next weekend to finalize those up because the book isn't officially published until later in the year. We've got the writing done, but the coding is still coming.
Mishra: Cool.
Nic: That's awesome. And I want to add to that I know from working with you that even if [the listener isn't] a Java developer, I definitely recommend picking it up and taking a read because the concepts, regardless of the language, are going to be solid and essential to anybody who is working with modern, cloud-based systems.
Daniel: Cheers, Nic, I appreciate that very much.
Nic: I'm looking forward to picking it up and grabbing a read. I might even pick up the early access on Safari this weekend.
Daniel: Awesome.
Mishra: I know what I'm getting for my Christmas gift.
Daniel: Thank you, Mishra. That's awesome.
Nic: Yeah, I'll download you the PDF, Mishra.
Mishra: Thanks, Nic.
Nic: Just before we go, I have one last question for you, and it's our most important question, and our most—well, it's completely not serious at all.
Daniel: Nice.
Nic: If you were a programming language from history—I'm talking about not something modern like Go or Rust or something like that; I'm thinking something historic like Java or something—w
hat would you be, and why?
Daniel: Oh, my gosh.
Mishra: That's hard.
Daniel: That's a genuinely hard question. But one thing that I really struggled with during PhD times was Prolog. Don't know if you ever bumped into Prolog. It's a list-derivative-type thing. I think I'd be Prolog because I like to think I'm fairly straightforward and simple on the outside, but kind of deep on the inside.
Nic: Oh, yeah.
Daniel: That's probably a bit of a deep answer, but Prolog was one of those things—it looked so simple, but to actually understand it took a lot of time. I think that's generally how I roll, as I might appear quite understandable, but actually when you get to know me, you realize there are multiple levels.
Nic: How about you, Mishra?
Mishra: Oh, man. That's a really hard question. I think something that I was introduced to really early on when I started using Windows and I loved: If you remember batch programs, simple batch programs that you could write. I think that was pretty elegant and interesting to me because you could do a lot of hacking. It was mostly writing scripts and stuff, but it was something that you could actually make a lot of use off, and actually make your life really easy on Windows. And I felt like that resonated with me as a person. It was simple to learn. You see what you get, basically. And I think that's one of the things I feel even today I look toward. That's why I like Go so much; basically, you're able to write things. But I'm not comparing batch programs with Go; let's just be clear. I'm just saying in terms of simplicity.
Nic: We just lost half our audience.
Daniel: Yeah.
Nic: Mishra compares batch to Go.
Mishra: Oh, my God. I just feel like batch programs were simple, and I think Go programs are pretty simple, too, when it comes to reading them. And I think that to me is really important.
Nic: Hot topic for the moment: What's your stance on generics?
Mishra: Oh, my God. Let's just move on.
Nic: Let's not lose the rest of the audience. Daniel, it's been an absolute pleasure. It's been wonderful speaking to you. Thank you so much for coming onto the show. And I hope our listeners really, really enjoyed this episode. Certainly one for the archives just for the amount of knowledge that you've shared with us, so thanks again. It's been a pleasure.
Daniel: Likewise.
Mishra: Thanks, Daniel.
Daniel: Appreciate both of you for the invite. It's been amazing. Keep up the good work, because I've been listening to the podcasts, and they've been really good fun. So appreciate both of you taking your time to do this. It's been awesome.
Nic: Thanks so much.
Mishra: You've been listening to HashiCast with your hosts Nic and Mishra. Today's guest has been Daniel Bryant from Datawire. Thanks for listening, and we'll see you next time.


text:
Crackers! Christmas debate turns into a load of baubles!
Join us now for a ringside seat at last week's special meeting of BTAC – called to discuss last year's Christmas lights fiasco.
Ringside might be the best possible word in the context – as by all accounts the lengthy session had more of the circus about it that the council chamber. Certainly there appeared to be no shortage of clowns.
Your guide to the evening is local businessman Darron Abbott, who is keen that the town be seen in a better light this Christmas – although once you've read his report, you may feel as we do that this appears increasingly unlikely
ednesday 22nd February, 6.30pm, the council chamber – finally we get to find out what went wrong with the with the 2016 Boston Christmas event. The meeting was open to the public, and was billed as not being held to apportion blame, but to move on to the future.
There were actually some members of the public in attendance – five of us in total, three people from a volunteer organisation, Councillor Jonathan Noble and me. Not bad for a meeting that had not really been publicised – not even a mention in the great Boston Bulletin.
The chairman, Councillor Nigel Welton, announced that the meeting would be recorded … was this a glimpse of a future of openness and accountability form the council or just to stop the council's officers being able to rewrite history to fit their own needs (but that is a story on its own.)
One thing that was a surprise at this point was the fact that no one from the infamous Town Team were present; instead, the Lincolnshire Chamber of Commerce supremo Simon Beardsley.
Once the pleasantries were out of the way I was allowed to ask my question, which centred on the ownership and the cost of the projectors used for the lighting display, as there seemed real confusion regarding this, and even the agenda notes for the meeting added to the confusion.
The reply came back, but it did not really answer the question other than to confirm that they had not been purchased but did not answer the lease or hire part.
"I then proceed with my supplementary question, in which I asked for a breakdown of how the £35,000 grant (from BTAC) had been spent.
The reply came back from an officer that the figures had been audited by an independent accountant and all was OK.
There was no offer to make these amounts public. I pressed again for a breakdown; it was then referred to Mr Beardsley who would not confirm that the breakdown would be made available to the public.
Why would they be reluctant to let us know how they had spent our money?
The meeting then continued mainly with statements from the members of the committee – most of which expressed surprise about the ownership of the equipment. Many of the committee had been given the impression that the equipment had been purchased and would be available for use at other times.
The other point that kept coming up was the fact that the actual display was nothing like that presented to committee by the Town Team.
There were statements from two senior committee members. Councillor Brian Rush said he had not become involved in the sub-committee to overlook the spending as he knew it would all go wrong and did not want to be part of it. Perhaps if he had been involved it would have meant he would have had to shoulder some of the responsibility.
Councillor Alison Austin apologised for not getting involved when perhaps she should have done – seeming to suggest she had not had time. Perhaps this was to ease her conscience for not having the time to help for the event but being able turn up and ensure she was in most of the photographs!
The funniest statement of the evening was his worshipfulness Woodliffe, who had seen nothing wrong with the event. It was marvellous, wonderful. I think he thought the Santa Claus present at the event was the real one.
The gnashing and wailing went on for well over an hour,. During this time to his credit the chairman did state that they were all guilty of neglecting to ensure the money had been spent wisely.
What was becoming apparent was that no one present was confident in allowing the Town Team or Chamber to be allowed to take on the event in the future.
Towards the end of this session, Councillor Yvonne Stevens spoke, being very critical of the Chamber's handing of the event.
When she finished she was approached by the council's monitoring officer and taken outside, which seemed very strange. She had not appeared to say anything controversial, so why would a council officer interfere, she was only there for guidance.
After a few minutes, Councillor Stevens returned to the chamber. The monitoring officer returned a few minutes later with an A4 sheet of paper and passed it to the chairman.
After reading what was on the paper, the chairman called a break in proceedings to allow refreshments.
The contents of the notes on this sheet of paper will never be known to us, the public, unless in the spirit of openness and accountability shown by Councillor Welton at the start of the meeting he would like to anonymously supply Boston Eye a copy! I am sure his identity would not be revealed.
Whilst most people's attentions were being diverted, getting cups of coffee and having a chat, at the other end of the chamber there were obvious discussions going on centred on the contents of the notes produced by the monitoring officer.
Was it the fact that no one wanted the chamber involved; but this is not what had been agreed in some kind of shoddy deal between the officers and chamber prior to the meeting?
The meeting then recommenced, with Simon Beardsley stating the town team had appointed a new chairman who was one of the town's largest retailers, but whom he declined to name.
They had already had meetings with Pescod Square about their lights and their lighting supplier, and that they had already started planning for the 2017 event.
This was not what a majority of those present wanted, so it was put to Beardsley as to whether the Town Team would like to be involved in community lead team. He did not like this. He "would have to go back to the Town Team and ask them as he could not speak on their behalf."
Why in hell was he at the meeting if he wasn't representing the Town Team? It seems that if the chamber were not running the show. They were not interested.
In the end, it was resolved that a committee be set up by March under the guidance of BTAC of residents and business, with the chamber if they wished and possibly the Federation of Small Businesses to run the events for the town.
It was also suggested that matched funding of up to £10.000 would be made available by BTAC and that any interested members of the public should contact Janette Collier from the council.
Then on Friday the council issued a press release stating people who are interested should contact the Chamber of Commerce in Lincoln, which was not what was agreed at the meeting!
A 'phone call to the Chamber of Commerce revealed that lady responsible does not work on Fridays, and would not be able to answer my questions as she was not going to be told she was responsible until Monday.
In its "account" of the meeting, the Worst Street website informed us that a new group had been formed to include something calling itself Boston More In Common Facebook Group.
On its Facebook page, the group itself asks "So...what is Boston More In Common? It is a group started by like-minded people to foster a sense of open friendliness between the rich tapestry of nationalities and cultures we are lucky to be home to in Boston, Lincolnshire. .. it is a place that we are trying to create to encourage interaction and integration ... to make us all realise that Boston is now 'Home' to many people, and we are all 'Bostonians' wherever we were born."
We wait with bated breath so see how this will translate into a stonking display of lights this year.
We hear that Worst Street is planning yet another of its pointless "Task and Finish" groups – this time to take a look at on impact of STP on the Pilgrim Hospital.
Until now STP was better known as a go-faster fuel additive – but this particular set of initials stands for Sustainability and Transformation Plan – a five year plan to improve quality and develop new models of care; improve health and well-being and the efficiency of services at the Pilgrim Hospital.
Task and finish groups may be established by a scrutiny committee to conduct an in-depth review of any service, policy or issue that affects the borough, which falls under the remit of the committee.
Given that Boston Borough Council seems incapable of running its own affairs, we can only wonder why it has the impudence to believe it can tell others how to manage theirs.
Its track record to date has not been impressive.
In recent years we have seen a report on the social impact of population change in Boston – which has been a source of considerable criticism, a report on the lamentably awful Boston Business "Improvement" District which saw recommendations that were completely ignored, and more recently a rural isolation task and finish group. Whether that achieved anything we have no idea.
Then of course there is the almost forgotten Prosperous Boston task and finish group, which we have dubbed the Preposterous Boston. By an ironic coincidence, as we dictated this sentence, our voice recognition software decided to call it the task "unfinished" group.
When the committee was established in 2015, Worst Street called it "one of the most in-depth studies into what makes Boston tick" aimed at making the town better for residents, shoppers and for those who work and visit here.
The group meets in secret, and does not publish minutes – but we are promised a final report will appear on the borough's website, which if they are running on schedule should be any time now.
Preposterous Boston's next update will be discussed on Wednesday a meeting of the Environment and Performance Committee – and a look through the report suggests that most of the things proposed are mere relatively minor aside from a plan for a food festival in September and the introduction of some Big Brother heavy-footedness.
We have no idea where the food festival idea has come from … but Spalding has staged something similar in previous years over two days – and this year is reviewing whether to repeat the exercise due to poor attendance figures.
There was also mention of a squad of "environmental enforcement" officers starting work through a third party organisation this week and a bid by Worst Street to get permission for its first RIPA surveillance operation using a covert camera, in an area subjected to continual fly tipping.
For those who don't know, RIPA is the Regulation of Investigatory Powers Act 2000 which controls the powers of public bodies to carry out surveillance and investigation, and the interception of communications – and was created to fight terrorism and serious crime.
What we've read so far is not particularly earth-shattering but we were pleased to note that the group has at least rattled the cages of the Head of Town Centre, Leisure, Events and Culture, the Town Centre Services Manager, the Principal Museum, Arts and Heritage Officer and the Partnerships and Sustainability Manager.
With all these big names getting involved, success surely cannot be far behind.
Also in the report, mention is made of concern at a damaged road sign on the A52 approaching the town, which has gone unrepaired for many months.
Whilst members agreed that it reflected badly on the town and needed to be straightened "it was not within Boston's remit to do so as it was County Highways responsibility."
Things will never improve whilst attitudes such as this survive.
Incidentally another damaged is on the A16 coming into town from Wyberton, and that has also been in its present state for many months as well. This is one of the "special" Boston signs, which we would hope the council can do something about.
Local websites have been awash this week with the snap below of our Police and Crime Commissioner, his deputy and a couple of other macho characters posing alongside a car that once belonged to a hare courser, and which has been seized and crushed.
Whilst we have no problem with depriving these people of their transport, we wonder whether the vehicles could be put to better use.
We are sure that charities, for example could use a four-wheel drive or two to help them collect donations, or various groups that try to help others.
If nothing else, could the vehicles not be auctioned, and the proceeds used to support good causes?
Whilst it's now unrecognisable, we wonder whether the vehicle in the photo was once a Vauxhall Courser?
Tonight's full meeting of the council will be discussing and voting on the proposed council tax increase for the year ahead. But don't expect any surprises, or last minute reprieves.
A reader who contacted Worst Street in the middle of last week with a query was told that it wasn't possible to access council tax information that day "because we're printing out the bills."
Our recent piece on the quality of local "newspapers" stirred some fond memories for Boston Eye reader Tony Eves, who e-mailed us to say:
our comments regarding the Boston Standard over the past few blogs have been spot on, and the latest reference to twelve reporters and a senior reporter dedicated to local government shows just how devalued a once respected newspaper has become
"That senior reporter you refer to is probably (I hope!) my father, Allan Eves. As a child I remember he used to attend all the council meetings, often until late at night, then spend hours writing the report either in the office in Wide Bargate or at home. He never seemed to express a personal opinion (until 'Spectator' many years later after retirement) but always appeared to know personally most of the local councillors and what they thought.
"As you say, in those days the council and the members were more accountable because there were real people looking at them, listening to them, talking to them and asking questions to their face.
"Sometimes, privately, he wasn't very complimentary but, publicly, it wasn't his place to be so.
"Going back even before those times I remember he used to go out reporting - knowing something happened but going out to find out more. I once sat in the car just past the (as was) Sibsey railway bridge one weekend while he talked to someone who had escaped a car accident in a drain.
"Earlier still one of his memories was of visiting the relatives of recently deceased gentleman to record the details of his life then being asked to go into the front room and have a look at the late lamented as he lay in his coffin on the table.
"Sad as it is a newspaper these days seems to rely on items being sent in to them from various social media sources then arranged on a computer generated page with few checks on grammar and spelling.
"And as an afterthought, my mother worked as a reader with Meg Comer for a few years at the then Lincolnshire Standard so there were plenty of checks then.
"Apart from the inevitable computer 'spellchecker' is there anything else these days?"
Finally, one of those delightful errors that slip unnoticed through the system. This comes from the Preposterous Boston report on the appointment of environmental enforcement officers and asks why their tasks cannot be combined with those of the traffic wardens.
Or perhaps it wasn't a slip of the pen after all!
You can write to us at email@example.com
E-mails will be treated in confidence and published anonymously if requested.
Our former blog is archived at: http://bostoneyelincolnshire.blogspot.com
We are on Twitter – visit @eye_boston

subset: commoncrawl


text:
Click here to download Podcast or
Click here to download off iTunes
The stores say it's time to think about school; supplies line the shelves, uniforms fill the racks, and back-to-school sales abound. That means it's time for moms to do a quick check in to make sure you're ready to kick off a fantastic year.
We're taking a look at the three key areas to consider when planning your year and ways to celebrate going back-to-school. Then we'll be joined by Kristin Beltaos and Daniella Knell who'll share tips for parents of children with food allergies. There are definitely things you must do before school begins to ensure a smooth start and a safe school year for your child. And when it comes to food allergies look to these two ladies to make sure you have the bases covered.
Then, in Christmas Corner, we'll be taking a look at the one thing you must think about at this time of year so you don't end up with huge stress in December.
Daniella Knell is an allergy educator, community advocate and mom of two children managing food, animal, medicinal and seasonal allergies. She provides adult and children's allergy awareness classes, national telephone workshops, product videos and one-on-one consulting.
Her goal is to 'alleviate food allergy fears one classroom at a time' through a positive, uplifting and partnering approach. The strong support networks and community advocates this provides are invaluable.
At Smart Allergy~Friendly Education Daniella will make your personal journey easier by realizing the vast resources available to you. Living with food allergies isn't the end of eating as you know it. It's the beginning of expanding your knowledge of the allergy friendly world around you.
Kristin Beltaos, M.A., owner of A Gift of Miles, offers coaching/consulting in the areas of stress, food allergies and reproductive challenges. Kristin struggled with infertility and multiple miscarriages and is now a mother to two beautiful boys, one who has multiple food allergies. She firmly believes that humor can diffuse and carry us through almost any situation and relishes being a positive influence to others with these same challenges. Kristin is the creator of a series of workshops focused on living life fully with food allergies. She also wrote and led the implementation of the first 504 Plan for food allergies in her son's school. She was honored as the November/December 2011 InSeason Mom. Prior to starting her business, Kristin was employed and consulted for a variety of companies in the areas of strategic planning and marketing. To learn more, visit her website at www.agiftofmiles.com, Facebook.com/agiftofmiles and Twitter: KristinBeltaos.Return to the Heart Filled Holidays Show Page
Link to Article


text:
During the past year gin has really started to become a hobby for me. That sounds like I've got a problem, but bear with me! I've read books about gin, gone to tasting evenings, my friends and I have held our own "Bring a Gin (or two) Day". I've tried many more gins in the past 12 months than I had in the past 12 years before.It's definitely become a thriving industry over the last decade with small batch, handcrafted gins popping up everywhere. My drinks cupboard has more or less become a gin cupboard now, and with even the supermarkets stocking a considerable range of interesting looking bottles it's hard to resist trying a new one each time I find myself down the aisle of temptation. I never realised how varied and different one gin could be from another, and it's no longer a case of just pairing a gin with a warm Schweppes tonic and an old slice of lemon. The tonics these days are far less sweet, with subtle flavourings that complement the botanicals in the gins. The garnishes call for an extensive herb garden (if I could manage to keep them alive, however green-fingered I am not!) or store cupboard stocks. I've even graduated to trying gin neat, but it has to be ice-cold. For drinking purposes I prefer my gin with tonic (around 1 part gin to 2 parts tonic for me), but on its own is the best way to first taste a gin.
I've rambled enough, let's crack on with our first gin review!
Wednesdays call for gin. There's no denying this. It may be once the kids have gone to bed and calm descends over the household. It may be as soon as you step through the door after a long day at work. It may even be a cheeky lunchtime gin (just the one, of course). Today was that day. A lunchtime gin day. Luckily I have colleagues with a keen interest in gin too and we escaped for a 'lunchtime meeting'. They had Monkey 47 behind the bar, and our choice was made.
Details: Monkey 47 is distilled in the Black Forest in Germany. The 47 in the name comes both from the number of botanicals that go into this gin, and the fact it's bottled at 47% ABV. The monkey comes from the gin's back story – the recipe is attributed to a British Air Force Commander who was stationed in Germany after the second world war and who helped rebuild Berlin Zoo, including sponsoring an egret monkey named Max (who can be found on the label).
Bottle: I'm sure you shouldn't judge a gin by its bottle but honestly, who doesn't? A good, solid, apothecary-style bottle made from brown glass, stoppered with a cork. As mentioned, an illustration of Max the monkey takes pride of place on the label. I like this style, and it certainly stands out from the majority of gin bottles – all clear glass with accents of blue. Top marks.
Serve: Served with a Mediterranean Fever Tree tonic, a slice of orange and sprig of rosemary to garnish – very nice, very summery looking. A regular premium tonic would also allow the gin to shine, this gin needs no helping hand in the flavour department (47 botanicals, remember?).
Nose: Herbal sweetness, with a warm, woody undertone.
Tasting notes: Rich, robust and well-rounded. A complex and unique gin, it's hard to pick out an over-arching flavour category. It starts citrus-y, giving way to plenty of spice, fruit and herb flavours. I cannot even begin to try to identify all 47 botanicals, every sip suggests something new as the gin develops. A description here seems almost irrelevant as this gin is nearly everything a gin could be. Smooth was my main takeaway, all the botanicals really work together rather than fighting for attention. Smooth with a capital SMOO.
Drink, buy or bin? At around £37 for 500ml Monkey 47 is firmly in the upper price bracket but I will definitely buy this as I really enjoyed it. Consider it a treat gin, rather than an everyday drink.
Monkey 47 can be found at several online retailers, including Masters of Malt.


text:
|Caroline Garborino
I felt the Allied Arts garden entrance made an intriguing metaphor for self-discovery. The event was enchanting! Thanks again for a fantasy paint out. Caroline
|Leslie Grimm
There was so much to choose from at this beautiful site. I chose one of the large urns near the entrance. A few years ago the urns were covered with ivy, but the head gardener, Kathleen, removed the ivy and landscaped around them. The big-leaf green plants are called Winter-blooming Vergenia (if I spelled that right). A few of them still had there dark pink blossoms even though winter was over. Leslie
|Anna Jacke
|Mary Paquet
"My third painting is of some of the lovely flowers near the fountain. I used quite a bit of color on these white flowers to suggest the shadows and texture of the petals."
Another great day of painting with friends. Mary


text:
http://www.nato.int on August 27, 2015, informed that Secretary General Jens Stoltenberg praised Georgia's strong commitment to the Alliance and inaugurated a new Training Centre during his first visit to Tbilisi in August. Excerpts below:
At a joint press conference with President Giorgi Margvelashvili, Mr. Stoltenberg described Georgia as "a strong and reliable contributor to our shared security". He thanked Georgia for its commitment to its relationship with the Alliance – "a relationship that is getting closer and deeper" – and for being the second largest contributor to NATO's Resolute Support mission in Afghanistan.
Welcoming Georgia's impressive democratic reforms, Mr. Stoltenberg called the country "an example in the region and beyond", saying its efforts demonstrate "commitment to being part of the Euro-Atlantic family" and help Georgia prepare for NATO membership. He said he looked forward to continued progress, saying "Euro-Atlantic integration is not an easy road; it takes time, determined efforts and real change." He stressed, "NATO will continue to stand by Georgia."
The Secretary General also condemned Russia's ongoing interference in Georgia, saying "Russia's latest actions to move the administrative boundary line of the South Ossetia region of Georgia are yet another breach of Russia's international commitments." He stressed that NATO Allies remain "fully committed" to Georgia's territorial integrity within its internationally recognized borders.
Together with President Margvelashvili, Prime Minister Irakli Garibashvili, Foreign Minister Tamar Beruchashvili and Defence Minister Tinatin Khidasheli, the Secretary General opened the new NATO-Georgia Joint Training and Evaluation Centre at the Krtsanisi Military Facility, where Georgian troops will train alongside Allies and partners.
"There is more Georgia in NATO and more NATO in Georgia," Stoltenberg said. "And all these efforts help Georgia move closer to your aspiration of NATO membership".
The Training Centre is part of the Substantial NATO-Georgia Package, agreed at the Wales Summit last year to help Georgia in its aspirations to NATO membership. NATO-Georgia cooperation over many years has strengthened Georgia's forces, allowing them to work with Allies more closely than ever before.
Comment: The new NATO training center in Georgia is a valuable contribution to closer links of that country to Europe and the West. Georgia's location at the interchange of global traderoutes makes it geostrategically of great importance. In face of the Russian occupation of Abkhazia and South Ossetia Western support for the return of these areas to Georgia is crucial as is support for closer links to Europe of Moldavia, Armenia and Azerbaijan.


text:
The Integrated E-Z Split Key Cup is an innovative drug test cup that eliminates the handling of urine and donor tampering, providing a unique and efficient approach for on-site drug testing. Utilizing the patented key activator allows you to begin testing at your convenience. This integrated drug test cup also has a reserve split-specimen for additional testing should confirmation be needed. Free Ground Shipping Every Day!
# DOA-21107-019 (COC, mAMP, AMP, THC, OP,I PCP, BAR, BZO, MTD, MDMA) Box of 25.
#DOA-1127-041: (COC, mAMP, AMP, THC, MOP, BAR, BZO, MTD, MDMA, OXY, PPX, BUP) Box of 25.
# DOA-2167-019 (COC, THC, OPI, mAMP, AMP, BZO) Box of 25.
# DJA-177-111: (COC AMP THC MOP OPI BZO mAMP) + AD (pH, Specific Gravity, Oxidant) Box of 25.
# I-DOA-197-041: (COC, THC, OPI, mAMP, AMP, PCP, BZO, BAR, PPX) 1 Box (25).
# I-DUE-187-071: (COC, THC, MOP, mAMP, AMP, PCP, BZO, OXY) 1 Box (25).
# I-DUD-197-014: (COC, THC, OPI, mAMP, AMP, PCP, BZO, BAR, MTD) + AD (OX SG PH NI GL CR) 1 Box (25).
# I-DUD-187-013: (COC, THC, OPI, mAMP, AMP, PCP, BZO, BAR) 1 Box (25).
# I-DUD-1107-012: ( COC, THC, OPI, mAMP, AMP, PCP, BZO, BAR, MTD, TCA) 1 Box (25).
# I-DUA-167-291: (COC, THC, OPI, OXY, mAMP, AMP) AD (OX SG pH). 1 Bod of 25 tests.
# I-DUA-167-022: (COC, THC, OPI, BZO, mAMP, AMP) 1 Box (25).
# I-DUA-157-034: (COC, THC, OPI, PCP, mAMP) + AD (OX, SG, pH) 1 Box (25).
EZ Key Cup Drug Test | Integrated Drug Test Cups at low prices!
Our products are easy to use drug tests with up to 99% accurate. When you purchase drug tests from us, you can have confidence that they have all been given U.S. FDA Clearance. With these test kits you will get what we in the industry call a procedure card, which is just test instructions that will not only show you how to administer these test kits but also how to read the test results, You can also read the package inserts and it will also tell you how to read the test results.
Are you tired of calling a customer support number and waiting on hold or trying to deal with an automated service just to get to a real person? When you call us, during a normal business day (M-F 8am-4pm CST), a real person answers, no automated service just real customer service people ready to help you with any orders or even questions about the products. Free Ground Same Day Shipping


text: The Nicol Scales & Measurement branch now open in Austin provides tailored programs to fit the calibration needs of our customers. Our EasyCal program handles a wide range of devices, industries and locations. As an ISO 17025 accredited calibration company, NSM's calibration services ensure that our testing methods meet or exceed the requirements of standards such as ISO/IEC 17025, ASTM, Handbook 44, and the Texas Department of Agriculture Weights and Measures. All of the standards used are traceable to NIST (National Institute of Standards and Technology). Download our EasyCal Brochure.


text:
In life there is much talk about the word call. Well, I should qualify that statement, in the United States of America, in certain Christian circles there is a lot of talk about call.
Where am I called?
To what am I called?
To whom am I called?
What if I'm not living out my call?
This is what I'm called to!
This is where I'm called!
I can't do that because I'm not called to those people.
I can't serve there, it's just not an issue I'm called to.
I have been known to throw these phrases around with the best of them. I have sought long and hard after call and location and work. I know the seemingly ever present strive to be obedient to the "call" placed within me.
While I've been in Spain something unexpected has happened for me. I have so incredibly easily adapted to the culture that many around me have said that it's like I was born Spanish. Some have wondered if there is call on my life to Spain because of how easily I've been able to connect with people. Just yesterday I was having a conversation with someone who alluded to me fitting in Spain so easily and wondered what God was going to do with that. Though it was not asked, I felt a question, do you think it's God calling you here?
Though I believe that God calls people. It is clear and it is something I've witnessed, but I think we've twisted it so that much of what our true call is has been lost along the way.
Call, in many ways, has become equal to career. I am doing that to which I am called, is a phrase often used when one describes their career of choice.
In Spain, I've only been asked one time what it is that I do for a living. Two months, many new friends and ONE TIME, I've been asked what I do. That is a far cry from the reality of life in the States. As a result, I myself have stopped wrestling significantly with call. Instead, I've taken time to do the good I know I'm to do. James talks about that in his letter in the later New Testament. He says, "To him who knows the good he is to do and does not do it, to him it is sin."
Today I wonder if we miss the good we are to do because we are so busy chasing call? I wonder if we miss the good we are to do, because we are chasing significance?
Yesterday I had a conversation with a friend where we talked about how easy it is to be and to know that every conversation has significance when outside the borders of the US. In two months, my striving for the perfect living out of call has ceased. I am not chasing after significance. I am, instead, realizing the significance I've been living.
As a church culture in the states we strive so much. I wonder if we miss the point? Have we missed discipleship and mission because we've been searching for personal significance? Have we missed Jesus because we've been searching for his call on our lives? Have we missed people and moments because we're looking for the largest impact possible?
I'm sure I have.
Last week I sat with a man who owns a restaurant that we frequent. We've become friends over my months here, though he speaks little english and I very little spanish. He had seen me meet with the students one on one and asked if we could do the same thing. So here I was, sitting in a coffee shop, having a conversation about life and our stories and I was able to ask questions prompting thoughts about his journey personally and spiritually. The conversation was powerful, even through a translator.
He left and I felt overwhelmed by the significance of the moment. I wasn't standing before a crowd. I wasn't training large numbers of people. I wasn't rescuing a thousand people from horrific life situations. No one in the coffee shop would have even taken notice. But this moment, these two hours were incredibly signifiant. These two hours actually have the potential of being a catalyst for life change in him and therefore life change for all with whom he comes in contact. Every [single] person who has a changed life, encouraged heart or new spirit alters eternity! Therefore, this moment, was incredibly significant to the world. And, this moment was the significant I was to give my life to.
This significance, this moment, was not tied to a career. It was not tied to a people group, culture or country. But it was the good I knew I was to do. He was the person in front of me to love, not because I'm called to full time mission's in Spain, but because as one who follows after the way of Jesus, I have no choice but to love, invest and listen to those around me.
Being a minister is not a call. Being a missionary is not a call. Being a friend is not a call. NO! And, I believe we must stop using our words to describe them as such. Being a minister, sharing the story of the good news, and loving people are a part of the identity we've been given as children of a good King and as such they are a part of the good we know we are to do.
I did not go on a mission trip to Spain. I simply came to Spain to live, to learn, to teach and because I am now here and the people around me are all Spanish and it is mine to love them, to serve them, to give to them, to tell them the story of good news and to minister. Because I'm called to Spain? NO! But simply because I'm here and because I'm a child of a good King who has given us the incredible opportunity to love and give to those around us, no matter where we happen to be.


text:
Pihanga Art Gallery
Pihanga Art Gallery is a co-operative of local artists from the Turangi area. We showcase a wide range of beautiful art – paintings, pottery, jewellery, sculpture, felting, harakeke - flax fibre art, photography and woodturning.
Described by one tourist as "the best art gallery in a small town", Pihanga Gallery is popular amongst locals and visitors alike and artworks from our gallery have found homes all around the world.
Monday to Friday 10 a.m. to 4 p.m.
Saturdays 9 a.m. – 3 p.m.
Sundays 11 a.m. to 2 p.m.


text:
watch 01:50
The Loop (Games)
Do you like this video?
Orichalcum Armor is one of the Hard Mode armor sets, craftable from Orichalcum that appears in the world after breaking a Demon Altar or Crimson Altar. (Orichalcum or Mythril will be generated once the player's second, fifth, eighth, etc. Altar is destroyed)
Parts of the Set
- 13 defense
- 6% increase chance of critical strike
- 10 defense
- 11% increase in movement speed
- 4 defense
- 18% increase chance of critical strike using magic
- Increases maximum mana by 80
- 7 defense
- 15% increase chance of critical strike using ranged weapons
- 8% increase in movement speed
- 19 defense
- 7% increased melee damage
- 7% increase in melee speed
- 7% increase in movement speed
Set Bonus: Every time an enemy is struck, a petal will quickly strike through the screen similarly to a falling star summoned by a Starfury, dealing 20-30 damage to all enemies it touches as well as illuminating the surrounding areas the projectile passes through briefly. It can only be triggered roughly once every second, and will be triggered even by passive damage sources (such as bees from a Honeycomb or the Thorns buff), but not damage from debuffs or itself.
Update Info
- Added to the game.
Community content is available under CC-BY-SA unless otherwise noted.


text:
Student Ambassadors assist DCTC with Student Life programming, college recruitment, retention and community relations. Student Ambassadors reach out to both future and current students to make them aware of DCTC's programs and services. Ambassadors promote student activities and campus involvement. They also assist with campus events. Visit the Student Life Center or contact Student Life staff for more information.
As a Student Ambassador, you will have an exclusive opportunity to work with DCTC students, faculty and staff while serving as a positive role model to our future and current students. Student Ambassadors can elect to work fall, spring and/or summer semesters.


text:
Marriott Bonvoy Home
Find & Reserve
1 Room: 1 Adult, /room 0 Children/room
(Age-based rates may be available)
Enter code
Deals & Packages
Meetings & Events
Innovative spaces. Inspired experiences. Personalised services.
Our Brands
About Marriott Bonvoy
Visit our vibrant and open lobby that has been recently remodeled.
Our bright and spacious guest rooms offer all of the convenience of home.
Enjoy our spacious king suite that offers a separate living and sleeping area.
Enjoy our spacious, warm and welcoming dining room that can accommodate small and large parties.
Enjoy our fresh hot breakfast to get your day started right, or simply relax in our warm and inviting lobby. Coffee and tea are always available. This is a great place to meet up with your associates or family.


text:
Monday we had another early wake-up call as we needed to be at the gates of Granada's Crown Jewel tourist sight, the Alhambra, by 8:15 a.m., due to our reserved ticket entry time to this historic hilltop Moorish complex.
So what is it?
The Alhambra is a Moorish style royal palace that was the last and the greatest of its kind.
During the Reconquista, Granada was the final city needed to be conquered by the Spanish Catholic kings. The last Moorish King of Spain, Boabdil, surrendered the city and the Alhambra in 1492 to end the Reconquista.
The complex is divided into four sections of which the Palacios Nazaries is by far the most impressive and would be our first stop.
The Moors loved tiles
Laurie and the ever present
Moorish wall art
Moorish wall art
No icons allowed.
A Ceiling
More Wall Art
There would be many
archways . . .
archways . . .
And fountains
Fountains and flowing water are a big part of the relaxing lifestyle that one would have found here 700 years ago.
Remember, the Moors ruled Spain during Europe's Dark Ages when most of Europe lived in filfthy conditions.
Moorish art, architecture and sanitation systems were light years ahead of their Catholic neighbors.
Heading into the Palacios Nazaries
Allah Be Praised
A Ceiling
Blue Stalactites
The Courtyard of the Myrtles
Wall Carvings Abound
A Ceiling
Just flat out impressive
How did they do this
without computers?
without computers?
A Ceiling
Wall Carving Close-up
Stalactite Arch
The Courtyard of the Lions
An Ornate Ceiling
The color blue symbolizes Heaven
Yep, another Ceiling
A Serene Garden
He lived here for one year, 1829.
He wrote a book, "Tales of the Alhambra," based on his experiences and stories he had heard during his time here.
The view of the Albayzín district
Wonderful Fountain
We have got to expel
these heathens from Spain!
these heathens from Spain!
We finished touring the Palacios Nazaries and could not come away more impressed.
To sum it up in just one word, WOW!!!
We still had three more parts of the Alhambra to visit so we headed next to the Alcazaba which is an empty old fort in ruins.
Entry Arch to the Alcazaba
Alcazaba Entry Arch
from the other side
from the other side
Since this part of the Alhambra is just ruins, we decided not to enter.
Instead we aimed for . . .
King Charles V's Palace
This was added to the Alhambra after the Reconquista and has none of the charm of the Palacios Nazaries.
Horse hitch
Tired lion still standing guard
against a return by the Moors
against a return by the Moors
Entering King Charles V's Palace
This palace had a huge courtyard
The Generalife Gardens
This was our fourth and final stop on the Alhambra tour. These gardens also house the Moorish kings' Summer Palace.
Water everywhere to
soothe the soul
soothe the soul
We visited the Alhambra for about five hours and were simply awed.
It was now time to wind our way back down the hill to the main streets of Granada and some nourishment.
It was now time to wind our way back down the hill to the main streets of Granada and some nourishment.
Walking the cobblestone path
called La Cuesta de los Chinos
called La Cuesta de los Chinos
Nice spot for a sreet lamp
Our waiter at a café on the
Paseo de los Tristes
Tapas, tapas, tapas . . .
Excellent Guitarist
We bought his five Euro CD.
Clam Shells
The clam shell is the symbol of the Camino de Santiago. St. James, Santiago in Spanish, is credited for helping the Catholics route the Moors.
His full title is San Santiago Matamoros, i.e., St James Moor Killer.
Nice ecumenical touch.
Not Real.
Colorful Moorish Cloth
I needed a break from
the oppresive heat!
the oppresive heat!
A strawberry slush was perfect!
For a small fee he will artfully
write your name in Arabic
write your name in Arabic
It was time to head back to the hotel for both a break from the heat and a siesta before going out for our evening paseo and a 10:00 p.m. dinner.
Not everybody is happy in
these tough economic times
these tough economic times
Someone painted a soccer ball
and the number 7 to this statue
and the number 7 to this statue
Isn't that David Villa's number?
Just another church
Another Camino de Santiago
An Excellent Street Band
We ate a fine dinner at this outdoor café on the Plaza de Bib-Rambla. In the background you see my table view of, in order from left to right, an ice cream shop, our restaurant and another ice cream shop.
That is just not right!
We went to the one on our
far right for dessert
And it was SO good for us too!
In the morning we will be boarding another train, this time bound for our final stop in Spain's Andalicia, the city of Córdoba and its much heralded Mezquita.


text:
hello, new member here, found this forum after searching my ctd in google. i'm very frustrated and don't know where else to ask, so any help would be very appreciated.
this is the crash log i got from .Net script framework, if there is a better way to check ctd please let me know
Unhandled native exception occurred at 0x7FF6F7AEAB51 (SkyrimSE.exe+136AB51) on thread 2900!
some explanation about the crash: (pasted from my reddit)
-always on save (autosave,fast travel,manual save). the ctd 100% happens wherever and whenever i save after the specific stage of the quest.
-in specific quest with quest follower, and always at the near end of the quest. happened when i bring other followers or alone.
so far, the quests are:
thief guild main quest speaking with silence follower:mercer. ctd on save after the poison scene
trinity restored follower:karliah and brynjolf. ctd on save after nocturnal scene
2. dawnguard quest touching the sky follower: serana. ctd on save after killing vyrthur
3. dawnstar quest waking nightmare (skull of corruption) follower: erandur. ctd on save after killing veren and thorek
companion questline with follower didn't cause ctd
usually when this happen i just load the last save then use command coc to somewhere and use setstage to clear the quest and i can save afterwards but i'm curious what caused it. i have usseep, using latest updated skse and engine fixes. i have googled all possible 'ctd on save' solution but none seems to fix this, any help would be appreciated